Hi all,
As you might know the markup improvements have been discussed at kernel summit:
https://lwn.net/Articles/662930/
Unfortunately it died in a bikeshed fest due to an alliance of people who think docs are useless and you should just read code and others who didn't know how to build the kerneldoc into something pretty. Oh well.
But I still want this, and Dave Airlie is ok with using it for drm kerneldoc as long as I maintain the support. Plan is to merge the first patch to adjust gpu.tmpl into drm properly, and keep everything else in topic/kerneldoc at
git://anongit.freedesktop.org/drm-intel topic/kerneldoc
If you want to build pretty docs just install asciidoc and base your drm documentation patches on top of drm-intel-nightly. That tree also includes all of Dave's tree. Alternatively pull in the above topic branch.
Note that asciidoc is strictly optional, but without it the docbook will look a bit bad since beyond paragraphs there won't be any additional formatting (like tables, quoting for sourcecode snippets or lists).
Intel also has an autobuilder that pushes latest drm-intel-nightly docs to
http://dri.freedesktop.org/docs/drm/
Cheers, Daniel
Daniel Vetter (2): scripts/kernel-doc: Use asciidoc instead of markdown drm: Enable markdown^Wasciidoc for gpu.tmpl
Danilo Cesar Lemes de Paula (3): drm/doc: Convert to markdown scripts/kernel-doc: Adding infrastructure for markdown support scripts/kernel-doc: Improve Markdown results
Documentation/DocBook/Makefile | 25 +++++--- Documentation/DocBook/gpu.tmpl | 86 -------------------------- drivers/gpu/drm/drm_modes.c | 12 ++-- drivers/gpu/drm/drm_modeset_lock.c | 14 ++--- drivers/gpu/drm/drm_prime.c | 16 ++--- drivers/gpu/drm/i915/i915_reg.h | 48 +++++++-------- scripts/docproc.c | 54 ++++++++++++----- scripts/kernel-doc | 120 ++++++++++++++++++++++++++++++++----- 8 files changed, 203 insertions(+), 172 deletions(-)
From: Danilo Cesar Lemes de Paula danilo.cesar@collabora.co.uk
DRM Docbook is now Markdown ready. This means its doc is able to use markdown text on it.
* Documentation/DocBook/drm.tmpl: Contains a table duplicated from drivers/gpu/drm/i915/i915_reg.h. This is not needed anymore
* drivers/gpu/drm/drm_modeset_lock.c: had a code example that used to look pretty bad on html. Fixed by using proper code markup.
* drivers/gpu/drm/drm_prime.c: Remove spaces between lines to make a proper markup list.
* drivers/gpu/drm/i915/i915_reg.h: Altought pandoc supports tables, it doesn't support table cell spanning. But we can use fixed-width for those special cases.
* include/drm/drm_vma_manager.h: Another code example that should be proper indented with four spaces.
v2 (Daniel): Adjust name to gpu.xml due to rename.
v3 (Daniel): Split out the actual enabling in the Makefile - this way we can merge the conversion, while just keeping the enabling in a drm-private tree.
Signed-off-by: Danilo Cesar Lemes de Paula danilo.cesar@collabora.co.uk (v1) Cc: Randy Dunlap rdunlap@infradead.org Cc: Daniel Vetter daniel.vetter@ffwll.ch Cc: Laurent Pinchart laurent.pinchart@ideasonboard.com Cc: Jonathan Corbet corbet@lwn.net Cc: Herbert Xu herbert@gondor.apana.org.au Cc: Stephan Mueller smueller@chronox.de Cc: Michal Marek mmarek@suse.cz Cc: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org Cc: intel-gfx intel-gfx@lists.freedesktop.org Cc: dri-devel dri-devel@lists.freedesktop.org Signed-off-by: Daniel Vetter daniel.vetter@intel.com --- Documentation/DocBook/gpu.tmpl | 86 -------------------------------------- drivers/gpu/drm/drm_modes.c | 12 +++--- drivers/gpu/drm/drm_modeset_lock.c | 14 +++---- drivers/gpu/drm/drm_prime.c | 16 +++---- drivers/gpu/drm/i915/i915_reg.h | 48 ++++++++++----------- 5 files changed, 42 insertions(+), 134 deletions(-)
diff --git a/Documentation/DocBook/gpu.tmpl b/Documentation/DocBook/gpu.tmpl index 201dcd3c2e9d..c05d7df3067d 100644 --- a/Documentation/DocBook/gpu.tmpl +++ b/Documentation/DocBook/gpu.tmpl @@ -4039,92 +4039,6 @@ int num_ioctls;</synopsis> <sect2> <title>DPIO</title> !Pdrivers/gpu/drm/i915/i915_reg.h DPIO - <table id="dpiox2"> - <title>Dual channel PHY (VLV/CHV/BXT)</title> - <tgroup cols="8"> - <colspec colname="c0" /> - <colspec colname="c1" /> - <colspec colname="c2" /> - <colspec colname="c3" /> - <colspec colname="c4" /> - <colspec colname="c5" /> - <colspec colname="c6" /> - <colspec colname="c7" /> - <spanspec spanname="ch0" namest="c0" nameend="c3" /> - <spanspec spanname="ch1" namest="c4" nameend="c7" /> - <spanspec spanname="ch0pcs01" namest="c0" nameend="c1" /> - <spanspec spanname="ch0pcs23" namest="c2" nameend="c3" /> - <spanspec spanname="ch1pcs01" namest="c4" nameend="c5" /> - <spanspec spanname="ch1pcs23" namest="c6" nameend="c7" /> - <thead> - <row> - <entry spanname="ch0">CH0</entry> - <entry spanname="ch1">CH1</entry> - </row> - </thead> - <tbody valign="top" align="center"> - <row> - <entry spanname="ch0">CMN/PLL/REF</entry> - <entry spanname="ch1">CMN/PLL/REF</entry> - </row> - <row> - <entry spanname="ch0pcs01">PCS01</entry> - <entry spanname="ch0pcs23">PCS23</entry> - <entry spanname="ch1pcs01">PCS01</entry> - <entry spanname="ch1pcs23">PCS23</entry> - </row> - <row> - <entry>TX0</entry> - <entry>TX1</entry> - <entry>TX2</entry> - <entry>TX3</entry> - <entry>TX0</entry> - <entry>TX1</entry> - <entry>TX2</entry> - <entry>TX3</entry> - </row> - <row> - <entry spanname="ch0">DDI0</entry> - <entry spanname="ch1">DDI1</entry> - </row> - </tbody> - </tgroup> - </table> - <table id="dpiox1"> - <title>Single channel PHY (CHV/BXT)</title> - <tgroup cols="4"> - <colspec colname="c0" /> - <colspec colname="c1" /> - <colspec colname="c2" /> - <colspec colname="c3" /> - <spanspec spanname="ch0" namest="c0" nameend="c3" /> - <spanspec spanname="ch0pcs01" namest="c0" nameend="c1" /> - <spanspec spanname="ch0pcs23" namest="c2" nameend="c3" /> - <thead> - <row> - <entry spanname="ch0">CH0</entry> - </row> - </thead> - <tbody valign="top" align="center"> - <row> - <entry spanname="ch0">CMN/PLL/REF</entry> - </row> - <row> - <entry spanname="ch0pcs01">PCS01</entry> - <entry spanname="ch0pcs23">PCS23</entry> - </row> - <row> - <entry>TX0</entry> - <entry>TX1</entry> - <entry>TX2</entry> - <entry>TX3</entry> - </row> - <row> - <entry spanname="ch0">DDI2</entry> - </row> - </tbody> - </tgroup> - </table> </sect2>
<sect2> diff --git a/drivers/gpu/drm/drm_modes.c b/drivers/gpu/drm/drm_modes.c index cd74a0953f42..9480464434cf 100644 --- a/drivers/gpu/drm/drm_modes.c +++ b/drivers/gpu/drm/drm_modes.c @@ -553,10 +553,10 @@ EXPORT_SYMBOL(drm_gtf_mode_complex); * drivers/video/fbmon.c * * Standard GTF parameters: - * M = 600 - * C = 40 - * K = 128 - * J = 20 + * M = 600 + * C = 40 + * K = 128 + * J = 20 * * Returns: * The modeline based on the GTF algorithm stored in a drm_display_mode object. @@ -1212,7 +1212,7 @@ EXPORT_SYMBOL(drm_mode_connector_list_update); * This uses the same parameters as the fb modedb.c, except for an extra * force-enable, force-enable-digital and force-disable bit at the end: * - * <xres>x<yres>[M][R][-<bpp>][@<refresh>][i][m][eDd] + * <xres>x<yres>[M][R][-<bpp>][@<refresh>][i][m][eDd] * * The intermediate drm_cmdline_mode structure is required to store additional * options from the command line modline like the force-enable/disable flag. @@ -1491,4 +1491,4 @@ int drm_mode_convert_umode(struct drm_display_mode *out,
out: return ret; -} \ No newline at end of file +} diff --git a/drivers/gpu/drm/drm_modeset_lock.c b/drivers/gpu/drm/drm_modeset_lock.c index 6675b1428410..7c9ca2381d78 100644 --- a/drivers/gpu/drm/drm_modeset_lock.c +++ b/drivers/gpu/drm/drm_modeset_lock.c @@ -40,17 +40,15 @@ * The basic usage pattern is to: * * drm_modeset_acquire_init(&ctx) - * retry: + * retry: * foreach (lock in random_ordered_set_of_locks) { - * ret = drm_modeset_lock(lock, &ctx) - * if (ret == -EDEADLK) { - * drm_modeset_backoff(&ctx); - * goto retry; - * } + * ret = drm_modeset_lock(lock, &ctx) + * if (ret == -EDEADLK) { + * drm_modeset_backoff(&ctx); + * goto retry; + * } * } - * * ... do stuff ... - * * drm_modeset_drop_locks(&ctx); * drm_modeset_acquire_fini(&ctx); */ diff --git a/drivers/gpu/drm/drm_prime.c b/drivers/gpu/drm/drm_prime.c index 9f935f55d74c..27aa7183b20b 100644 --- a/drivers/gpu/drm/drm_prime.c +++ b/drivers/gpu/drm/drm_prime.c @@ -313,19 +313,15 @@ static const struct dma_buf_ops drm_gem_prime_dmabuf_ops = { * * Export callbacks: * - * - @gem_prime_pin (optional): prepare a GEM object for exporting - * - * - @gem_prime_get_sg_table: provide a scatter/gather table of pinned pages - * - * - @gem_prime_vmap: vmap a buffer exported by your driver - * - * - @gem_prime_vunmap: vunmap a buffer exported by your driver - * - * - @gem_prime_mmap (optional): mmap a buffer exported by your driver + * * @gem_prime_pin (optional): prepare a GEM object for exporting + * * @gem_prime_get_sg_table: provide a scatter/gather table of pinned pages + * * @gem_prime_vmap: vmap a buffer exported by your driver + * * @gem_prime_vunmap: vunmap a buffer exported by your driver + * * @gem_prime_mmap (optional): mmap a buffer exported by your driver * * Import callback: * - * - @gem_prime_import_sg_table (import): produce a GEM object from another + * * @gem_prime_import_sg_table (import): produce a GEM object from another * driver's scatter/gather table */
diff --git a/drivers/gpu/drm/i915/i915_reg.h b/drivers/gpu/drm/i915/i915_reg.h index bc7b8faba84d..9ebf03230ac0 100644 --- a/drivers/gpu/drm/i915/i915_reg.h +++ b/drivers/gpu/drm/i915/i915_reg.h @@ -804,31 +804,31 @@ enum skl_disp_power_wells { * * Note: DDI0 is digital port B, DD1 is digital port C, and DDI2 is * digital port D (CHV) or port A (BXT). - */ -/* - * Dual channel PHY (VLV/CHV/BXT) - * --------------------------------- - * | CH0 | CH1 | - * | CMN/PLL/REF | CMN/PLL/REF | - * |---------------|---------------| Display PHY - * | PCS01 | PCS23 | PCS01 | PCS23 | - * |-------|-------|-------|-------| - * |TX0|TX1|TX2|TX3|TX0|TX1|TX2|TX3| - * --------------------------------- - * | DDI0 | DDI1 | DP/HDMI ports - * --------------------------------- * - * Single channel PHY (CHV/BXT) - * ----------------- - * | CH0 | - * | CMN/PLL/REF | - * |---------------| Display PHY - * | PCS01 | PCS23 | - * |-------|-------| - * |TX0|TX1|TX2|TX3| - * ----------------- - * | DDI2 | DP/HDMI port - * ----------------- + * + * Dual channel PHY (VLV/CHV/BXT) + * --------------------------------- + * | CH0 | CH1 | + * | CMN/PLL/REF | CMN/PLL/REF | + * |---------------|---------------| Display PHY + * | PCS01 | PCS23 | PCS01 | PCS23 | + * |-------|-------|-------|-------| + * |TX0|TX1|TX2|TX3|TX0|TX1|TX2|TX3| + * --------------------------------- + * | DDI0 | DDI1 | DP/HDMI ports + * --------------------------------- + * + * Single channel PHY (CHV/BXT) + * ----------------- + * | CH0 | + * | CMN/PLL/REF | + * |---------------| Display PHY + * | PCS01 | PCS23 | + * |-------|-------| + * |TX0|TX1|TX2|TX3| + * ----------------- + * | DDI2 | DP/HDMI port + * ----------------- */ #define DPIO_DEVFN 0
From: Danilo Cesar Lemes de Paula danilo.cesar@collabora.co.uk
Markdown support is given by calling an external tool, pandoc, for all highlighted text on kernel-doc.
Pandoc converts Markdown text to proper Docbook tags, which will be later translated to pdf, html or other targets.
This adds the capability of adding human-readle text highlight (bold, underline, etc), bullet and numbered lists, simple tables, fixed-width text (including asciiart), requiring minimal changes to current documentation.
At this moment, pandoc is totally optional. Docbooks ready for markdown should be added to the MARKDOWNREADY variable inside the Makefile. In case the developer doesn't have pandoc installed, Make will throw a warning and the documentation build will continue, generating simple Documentation without the features brought by pandoc.
Signed-off-by: Danilo Cesar Lemes de Paula danilo.cesar@collabora.co.uk Cc: Randy Dunlap rdunlap@infradead.org Cc: Daniel Vetter daniel.vetter@ffwll.ch Cc: Laurent Pinchart laurent.pinchart@ideasonboard.com Cc: Jonathan Corbet corbet@lwn.net Cc: Herbert Xu herbert@gondor.apana.org.au Cc: Stephan Mueller smueller@chronox.de Cc: Michal Marek mmarek@suse.cz Cc: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org Cc: intel-gfx intel-gfx@lists.freedesktop.org Cc: dri-devel dri-devel@lists.freedesktop.org --- Documentation/DocBook/Makefile | 25 +++++++++++----- scripts/docproc.c | 54 ++++++++++++++++++++++++---------- scripts/kernel-doc | 66 ++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 119 insertions(+), 26 deletions(-)
diff --git a/Documentation/DocBook/Makefile b/Documentation/DocBook/Makefile index 91f6d89bb19f..246ad38550e5 100644 --- a/Documentation/DocBook/Makefile +++ b/Documentation/DocBook/Makefile @@ -17,6 +17,8 @@ DOCBOOKS := z8530book.xml device-drivers.xml \ tracepoint.xml gpu.xml media_api.xml w1.xml \ writing_musb_glue_layer.xml crypto-API.xml iio.xml
+MARKDOWNREADY := + include Documentation/DocBook/media/Makefile
### @@ -87,18 +89,23 @@ XMLTOFLAGS += --skip-validation # The following rules are used to generate the .xml documentation # required to generate the final targets. (ps, pdf, html). quiet_cmd_docproc = DOCPROC $@ - cmd_docproc = SRCTREE=$(srctree)/ $(DOCPROC) doc $< >$@ + cmd_docproc = SRCTREE=$(srctree)/ $(DOCPROC) doc $< define rule_docproc - set -e; \ - $(if $($(quiet)cmd_$(1)),echo ' $($(quiet)cmd_$(1))';) \ - $(cmd_$(1)); \ - ( \ - echo 'cmd_$@ := $(cmd_$(1))'; \ - echo $@: `SRCTREE=$(srctree) $(DOCPROC) depend $<`; \ + set -e; \ + USEMARKDOWN=""; \ + FILE=`basename $@`; \ + [[ "$(MARKDOWNREADY)" =~ "$${FILE}" ]] && USEMARKDOWN="-use-markdown"; \ + $(if $($(quiet)cmd_$(1)),echo ' $($(quiet)cmd_$(1))';) \ + $(cmd_$(1)) $$USEMARKDOWN >$@; \ + ( \ + echo 'cmd_$@ := $(cmd_$(1))'; \ + echo $@: `SRCTREE=$(srctree) $(DOCPROC) depend $<`; \ ) > $(dir $@).$(notdir $@).cmd endef
%.xml: %.tmpl $(KERNELDOC) $(DOCPROC) $(KERNELDOCXMLREF) FORCE + @(which pandoc > /dev/null 2>&1) || \ + (echo "*** To get propper documentation you need to install pandoc ***";) $(call if_changed_rule,docproc)
# Tell kbuild to always build the programs @@ -109,6 +116,10 @@ notfoundtemplate = echo "*** You have to install docbook-utils or xmlto ***"; \ db2xtemplate = db2TYPE -o $(dir $@) $< xmltotemplate = xmlto TYPE $(XMLTOFLAGS) -o $(dir $@) $<
+ifneq ($(shell which pandoc >/dev/null 2>&1 && echo found),found) + MARKDOWNREADY := ""; +endif + # determine which methods are available ifeq ($(shell which db2ps >/dev/null 2>&1 && echo found),found) use-db2x = db2x diff --git a/scripts/docproc.c b/scripts/docproc.c index e267e621431a..7c6b225a6a50 100644 --- a/scripts/docproc.c +++ b/scripts/docproc.c @@ -73,12 +73,15 @@ FILELINE * docsection; #define NOFUNCTION "-nofunction" #define NODOCSECTIONS "-no-doc-sections" #define SHOWNOTFOUND "-show-not-found" +#define USEMARKDOWN "-use-markdown"
static char *srctree, *kernsrctree;
static char **all_list = NULL; static int all_list_len = 0;
+static int use_markdown = 0; + static void consume_symbol(const char *sym) { int i; @@ -95,10 +98,11 @@ static void consume_symbol(const char *sym)
static void usage (void) { - fprintf(stderr, "Usage: docproc {doc|depend} file\n"); + fprintf(stderr, "Usage: docproc {doc|depend} [-use-markdown] file\n"); fprintf(stderr, "Input is read from file.tmpl. Output is sent to stdout\n"); fprintf(stderr, "doc: frontend when generating kernel documentation\n"); fprintf(stderr, "depend: generate list of files referenced within file\n"); + fprintf(stderr, "-use-markdown: pass -use-markdown to kernel-doc\n"); fprintf(stderr, "Environment variable SRCTREE: absolute path to sources.\n"); fprintf(stderr, " KBUILD_SRC: absolute path to kernel source tree.\n"); } @@ -257,12 +261,15 @@ static void docfunctions(char * filename, char * type)
for (i=0; i <= symfilecnt; i++) symcnt += symfilelist[i].symbolcnt; - vec = malloc((2 + 2 * symcnt + 3) * sizeof(char *)); + vec = malloc((2 + 2 * symcnt + 4) * sizeof(char *)); if (vec == NULL) { perror("docproc: "); exit(1); } vec[idx++] = KERNELDOC; + if (use_markdown) + vec[idx++] = USEMARKDOWN; + vec[idx++] = DOCBOOK; vec[idx++] = NODOCSECTIONS; for (i=0; i < symfilecnt; i++) { @@ -294,6 +301,9 @@ static void singfunc(char * filename, char * line) int i, idx = 0; int startofsym = 1; vec[idx++] = KERNELDOC; + if (use_markdown) + vec[idx++] = USEMARKDOWN; + vec[idx++] = DOCBOOK; vec[idx++] = SHOWNOTFOUND;
@@ -328,8 +338,9 @@ static void singfunc(char * filename, char * line) static void docsect(char *filename, char *line) { /* kerneldoc -docbook -show-not-found -function "section" file NULL */ - char *vec[7]; + char *vec[8]; char *s; + int idx = 0;
for (s = line; *s; s++) if (*s == '\n') @@ -342,30 +353,37 @@ static void docsect(char *filename, char *line) consume_symbol(s); free(s);
- vec[0] = KERNELDOC; - vec[1] = DOCBOOK; - vec[2] = SHOWNOTFOUND; - vec[3] = FUNCTION; - vec[4] = line; - vec[5] = filename; - vec[6] = NULL; + vec[idx++] = KERNELDOC; + if (use_markdown) + vec[idx++] = USEMARKDOWN; + + vec[idx++] = DOCBOOK; + vec[idx++] = SHOWNOTFOUND; + vec[idx++] = FUNCTION; + vec[idx++] = line; + vec[idx++] = filename; + vec[idx] = NULL; exec_kernel_doc(vec); }
static void find_all_symbols(char *filename) { - char *vec[4]; /* kerneldoc -list file NULL */ + char *vec[5]; /* kerneldoc -list file NULL */ pid_t pid; int ret, i, count, start; char real_filename[PATH_MAX + 1]; int pipefd[2]; char *data, *str; size_t data_len = 0; + int idx = 0; + + vec[idx++] = KERNELDOC; + if (use_markdown) + vec[idx++] = USEMARKDOWN;
- vec[0] = KERNELDOC; - vec[1] = LIST; - vec[2] = filename; - vec[3] = NULL; + vec[idx++] = LIST; + vec[idx++] = filename; + vec[idx] = NULL;
if (pipe(pipefd)) { perror("pipe"); @@ -509,7 +527,7 @@ int main(int argc, char *argv[]) kernsrctree = getenv("KBUILD_SRC"); if (!kernsrctree || !*kernsrctree) kernsrctree = srctree; - if (argc != 3) { + if (argc < 3 || argc > 4) { usage(); exit(1); } @@ -521,6 +539,10 @@ int main(int argc, char *argv[]) exit(2); }
+ if (argc == 4 && strcmp("-use-markdown", argv[3]) == 0) { + use_markdown = 1; + } + if (strcmp("doc", argv[1]) == 0) { /* Need to do this in two passes. * First pass is used to collect all symbols exported diff --git a/scripts/kernel-doc b/scripts/kernel-doc index 638a38e1b419..28737053395a 100755 --- a/scripts/kernel-doc +++ b/scripts/kernel-doc @@ -1,6 +1,7 @@ #!/usr/bin/perl -w
use strict; +use IPC::Open3;
## Copyright (c) 1998 Michael Zucchi, All Rights Reserved ## ## Copyright (C) 2000, 1 Tim Waugh twaugh@redhat.com ## @@ -282,6 +283,7 @@ if ($#ARGV == -1) {
my $kernelversion; my $dohighlight = ""; +my $use_markdown = 0;
my $verbose = 0; my $output_mode = "man"; @@ -424,6 +426,8 @@ while ($ARGV[0] =~ m/^-(.*)/) { $function_only = 2; $function = shift @ARGV; $function_table{$function} = 1; + } elsif ($cmd eq "-use-markdown") { + $use_markdown = 1; } elsif ($cmd eq "-v") { $verbose = 1; } elsif (($cmd eq "-h") || ($cmd eq "--help")) { @@ -442,6 +446,7 @@ sub usage { print " [ -no-doc-sections ]\n"; print " [ -function funcname [ -function funcname ...] ]\n"; print " [ -nofunction funcname [ -nofunction funcname ...] ]\n"; + print " [ -use-markdown ]\n"; print " [ -v ]\n"; print " c source file(s) > outputfile\n"; print " -v : verbose output, more warnings & other info listed\n"; @@ -515,6 +520,49 @@ sub dump_doc_section { } }
+sub markdown_to_docbook { + my $orig_content = $_[0]; + + my $pid = open3( *CHLD_IN, *CHLD_OUT, *CHLD_ERR, "pandoc --columns=80 -f markdown -t docbook" ); + + print CHLD_IN "$orig_content"; + close(CHLD_IN); + + waitpid($pid, 0); + + my $content = ""; + chomp(my @lines = <CHLD_OUT>); + foreach my $line (@lines) { + $content .= $line . "\n"; + } + close(CHLD_OUT); + close(CHLD_ERR); + + # pandoc insists in adding Main <para></para>, we should remove them. + $content =~ s:\A\s*<para>\s*\n(.*)\n</para>\Z$:$1:egsm; + + return $content; +} + +# Markdown->Docbook conversion by pandoc requires unescaped text +# Kernel-doc converts every & to "&", we need to convert it back. +sub markdown_unescape +{ + my $text = shift; + my @lines = split /\n/, $text; + + my @result; + foreach my $line (@lines) { + if ( $line =~ m /^ /s ) { + $line =~ s:&:&:gs + } + push @result, $line; + } + + return join "\n",@result; + +} + ## # output function # @@ -541,11 +589,19 @@ sub output_highlight { $contents = local_unescape($contents); # convert data read & converted thru xml_escape() into &xyz; format: $contents =~ s/\\\/&/g; + + if ($use_markdown) { + $contents = markdown_unescape($contents); + } } + # print STDERR "contents b4:$contents\n"; eval $dohighlight; die $@ if $@; # print STDERR "contents af:$contents\n"; + if ($use_markdown) { + $contents = markdown_to_docbook($contents); + }
# strip whitespaces when generating html5 if ($output_mode eq "html5") { @@ -553,7 +609,8 @@ sub output_highlight { $contents =~ s/\s+$//; } foreach $line (split "\n", $contents) { - if (! $output_preformatted) { + if (! $output_preformatted && + !($use_markdown && $line =~ m /^ /s)) { $line =~ s/^\s*//; } if ($line eq ""){ @@ -974,7 +1031,9 @@ sub output_section_xml(%) { print "<refsect1>\n"; print "<title>$section</title>\n"; if ($section =~ m/EXAMPLE/i) { - print "<informalexample><programlisting>\n"; + print "<informalexample>\n"; + # programlisting is already included by pandoc + print "<programlisting>\n" unless $use_markdown; $output_preformatted = 1; } else { print "<para>\n"; @@ -982,7 +1041,8 @@ sub output_section_xml(%) { output_highlight($args{'sections'}{$section}); $output_preformatted = 0; if ($section =~ m/EXAMPLE/i) { - print "</programlisting></informalexample>\n"; + print "</programlisting>\n" unless $use_markdown; + print "</informalexample>\n"; } else { print "</para>\n"; }
From: Danilo Cesar Lemes de Paula danilo.cesar@collabora.co.uk
Using pandoc as the Markdown engine cause some minor side effects as pandoc includes main <para> tags for almost everything. Original Markdown support approach removes those main tags, but it caused some inconsistencies when that tag is not the main one, like: <something>..</something> <para>...</para>
As kernel-doc was already including a <para> tag, it causes the presence of double <para> tags (<para><para>), which is not supported by DocBook spec.
Html target gets away with it, so it causes no harm, although other targets might not be so lucky (pdf as example).
We're now delegating the inclusion of the main <para> tag to pandoc only, as it knows when it's necessary or not.
That behavior causes a corner case, the only situation where we're certainly that <para> is not needed, which is the <refpurpose> content. For those cases, we're using a $output_markdown_nopara = 1 control var.
Signed-off-by: Danilo Cesar Lemes de Paula danilo.cesar@collabora.co.uk Cc: Randy Dunlap rdunlap@infradead.org Cc: Daniel Vetter daniel.vetter@ffwll.ch Cc: Laurent Pinchart laurent.pinchart@ideasonboard.com Cc: Jonathan Corbet corbet@lwn.net Cc: Herbert Xu herbert@gondor.apana.org.au Cc: Stephan Mueller smueller@chronox.de Cc: Michal Marek mmarek@suse.cz Cc: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org Cc: intel-gfx intel-gfx@lists.freedesktop.org Cc: dri-devel dri-devel@lists.freedesktop.org Cc: Graham Whaley graham.whaley@linux.intel.com --- scripts/kernel-doc | 48 ++++++++++++++++++++++++++++++++++-------------- 1 file changed, 34 insertions(+), 14 deletions(-)
diff --git a/scripts/kernel-doc b/scripts/kernel-doc index 28737053395a..e01e74f15a22 100755 --- a/scripts/kernel-doc +++ b/scripts/kernel-doc @@ -288,6 +288,7 @@ my $use_markdown = 0; my $verbose = 0; my $output_mode = "man"; my $output_preformatted = 0; +my $output_markdown_nopara = 0; my $no_doc_sections = 0; my @highlights = @highlights_man; my $blankline = $blankline_man; @@ -538,8 +539,11 @@ sub markdown_to_docbook { close(CHLD_OUT); close(CHLD_ERR);
- # pandoc insists in adding Main <para></para>, we should remove them. - $content =~ s:\A\s*<para>\s*\n(.*)\n</para>\Z$:$1:egsm; + if ($output_markdown_nopara) { + # pandoc insists in adding Main <para></para>, sometimes we + # want to remove them. + $content =~ s:\A\s*<para>\s*\n(.*)\n</para>\Z$:$1:egsm; + }
return $content; } @@ -614,7 +618,7 @@ sub output_highlight { $line =~ s/^\s*//; } if ($line eq ""){ - if (! $output_preformatted) { + if (! $output_preformatted && ! $use_markdown) { print $lineprefix, local_unescape($blankline); } } else { @@ -1035,7 +1039,7 @@ sub output_section_xml(%) { # programlisting is already included by pandoc print "<programlisting>\n" unless $use_markdown; $output_preformatted = 1; - } else { + } elsif (! $use_markdown) { print "<para>\n"; } output_highlight($args{'sections'}{$section}); @@ -1043,7 +1047,7 @@ sub output_section_xml(%) { if ($section =~ m/EXAMPLE/i) { print "</programlisting>\n" unless $use_markdown; print "</informalexample>\n"; - } else { + } elsif (! $use_markdown) { print "</para>\n"; } print "</refsect1>\n"; @@ -1075,7 +1079,9 @@ sub output_function_xml(%) { print " <refname>" . $args{'function'} . "</refname>\n"; print " <refpurpose>\n"; print " "; + $output_markdown_nopara = 1; output_highlight ($args{'purpose'}); + $output_markdown_nopara = 0; print " </refpurpose>\n"; print "</refnamediv>\n";
@@ -1113,10 +1119,12 @@ sub output_function_xml(%) { $parameter_name =~ s/[.*//;
print " <varlistentry>\n <term><parameter>$parameter</parameter></term>\n"; - print " <listitem>\n <para>\n"; + print " <listitem>\n"; + print " <para>\n" unless $use_markdown; $lineprefix=" "; output_highlight($args{'parameterdescs'}{$parameter_name}); - print " </para>\n </listitem>\n </varlistentry>\n"; + print " </para>\n" unless $use_markdown; + print " </listitem>\n </varlistentry>\n"; } print " </variablelist>\n"; } else { @@ -1152,7 +1160,9 @@ sub output_struct_xml(%) { print " <refname>" . $args{'type'} . " " . $args{'struct'} . "</refname>\n"; print " <refpurpose>\n"; print " "; + $output_markdown_nopara = 1; output_highlight ($args{'purpose'}); + $output_markdown_nopara = 0; print " </refpurpose>\n"; print "</refnamediv>\n";
@@ -1205,9 +1215,11 @@ sub output_struct_xml(%) { ($args{'parameterdescs'}{$parameter_name} ne $undescribed) || next; print " <varlistentry>"; print " <term>$parameter</term>\n"; - print " <listitem><para>\n"; + print " <listitem>\n"; + print " <para>\n" unless $use_markdown; output_highlight($args{'parameterdescs'}{$parameter_name}); - print " </para></listitem>\n"; + print " </para>\n" unless $use_markdown; + print " </listitem>\n"; print " </varlistentry>\n"; } print " </variablelist>\n"; @@ -1246,7 +1258,9 @@ sub output_enum_xml(%) { print " <refname>enum " . $args{'enum'} . "</refname>\n"; print " <refpurpose>\n"; print " "; + $output_markdown_nopara = 1; output_highlight ($args{'purpose'}); + $output_markdown_nopara = 0; print " </refpurpose>\n"; print "</refnamediv>\n";
@@ -1276,9 +1290,11 @@ sub output_enum_xml(%) {
print " <varlistentry>"; print " <term>$parameter</term>\n"; - print " <listitem><para>\n"; + print " <listitem>\n"; + print " <para>\n" unless $use_markdown; output_highlight($args{'parameterdescs'}{$parameter_name}); - print " </para></listitem>\n"; + print " </para>\n" unless $use_markdown; + print " </listitem>\n"; print " </varlistentry>\n"; } print " </variablelist>\n"; @@ -1344,14 +1360,14 @@ sub output_blockhead_xml(%) { if ($section =~ m/EXAMPLE/i) { print "<example><para>\n"; $output_preformatted = 1; - } else { + } elsif (! $use_markdown) { print "<para>\n"; } output_highlight($args{'sections'}{$section}); $output_preformatted = 0; if ($section =~ m/EXAMPLE/i) { print "</para></example>\n"; - } else { + } elsif (! $use_markdown) { print "</para>"; } if (!$args{'content-only'}) { @@ -2721,7 +2737,11 @@ sub process_file($) { { if ( $1 eq "" ) { - $contents .= $blankline; + if ($use_markdown) { + $contents .= "\n"; + } else { + $contents .= $blankline; + } } else {
By popular demand.
This needs some adjustment/fixups after feeding snippets to asciidoc since compared to markdown asciidown escapes xml markup and doesn't just let it through.
The other noticeable change is that build times increase a lot - we need to launch the markup process per-snippet, there's a few thousand of them and asciidoc (python) has a substantial higher overhead per invocation than pandoc (haskell).
Cc: Danilo Cesar Lemes de Paula danilo.cesar@collabora.co.uk Cc: Thomas Wood thomas.wood@intel.com Cc: Jonathan Corbet corbet@lwn.net Signed-off-by: Daniel Vetter daniel.vetter@intel.com --- Documentation/DocBook/Makefile | 6 +++--- scripts/kernel-doc | 12 +++++++++++- 2 files changed, 14 insertions(+), 4 deletions(-)
diff --git a/Documentation/DocBook/Makefile b/Documentation/DocBook/Makefile index 246ad38550e5..5335955c0de5 100644 --- a/Documentation/DocBook/Makefile +++ b/Documentation/DocBook/Makefile @@ -104,8 +104,8 @@ define rule_docproc endef
%.xml: %.tmpl $(KERNELDOC) $(DOCPROC) $(KERNELDOCXMLREF) FORCE - @(which pandoc > /dev/null 2>&1) || \ - (echo "*** To get propper documentation you need to install pandoc ***";) + @(which asciidoc > /dev/null 2>&1) || \ + (echo "*** To get propper documentation you need to install asciidoc ***";) $(call if_changed_rule,docproc)
# Tell kbuild to always build the programs @@ -116,7 +116,7 @@ notfoundtemplate = echo "*** You have to install docbook-utils or xmlto ***"; \ db2xtemplate = db2TYPE -o $(dir $@) $< xmltotemplate = xmlto TYPE $(XMLTOFLAGS) -o $(dir $@) $<
-ifneq ($(shell which pandoc >/dev/null 2>&1 && echo found),found) +ifneq ($(shell which asciidoc >/dev/null 2>&1 && echo found),found) MARKDOWNREADY := ""; endif
diff --git a/scripts/kernel-doc b/scripts/kernel-doc index e01e74f15a22..c8eed5299a4b 100755 --- a/scripts/kernel-doc +++ b/scripts/kernel-doc @@ -524,7 +524,7 @@ sub dump_doc_section { sub markdown_to_docbook { my $orig_content = $_[0];
- my $pid = open3( *CHLD_IN, *CHLD_OUT, *CHLD_ERR, "pandoc --columns=80 -f markdown -t docbook" ); + my $pid = open3( *CHLD_IN, *CHLD_OUT, *CHLD_ERR, "asciidoc --no-header-footer --backend=docbook45 -" );
print CHLD_IN "$orig_content"; close(CHLD_IN); @@ -605,6 +605,16 @@ sub output_highlight { # print STDERR "contents af:$contents\n"; if ($use_markdown) { $contents = markdown_to_docbook($contents); + + # Compared to markdown asciidoc doesn't let through arbitrary xml + # markup. We need to un-escape the kerneldoc markup for functions, + # structures, ... + $contents =~ s/<quote>(\S*)</quote>/<quote>$1</quote>/g; + $contents =~ s/<constant>(\S*)</constant>/<constant>$1</constant>/g; + $contents =~ s/<structname>(\S*)</structname>/<structname>$1</structname>/g; + $contents =~ s/<parameter>(\S*)</parameter>/<parameter>$1</parameter>/g; + $contents =~ s/<function>(\S*)</function>/<function>$1</function>/g; + $contents =~ s/<envar>(\S*)</envar>/<envar>$1</envar>/g; }
# strip whitespaces when generating html5
By popular demand.
This needs some adjustment/fixups after feeding snippets to asciidoc since compared to markdown asciidown escapes xml markup and doesn't just let it through.
The other noticeable change is that build times increase a lot - we need to launch the markup process per-snippet, there's a few thousand of them and asciidoc (python) has a substantial higher overhead per invocation than pandoc (haskell).
v2: More fine-tuning:
- Use unixe newlines, not the default dos ones. Only results in ugliness in the intermediate gpu.xml, but still.
- Resurrect the hack to remove paragraphs for the one-line references. Like markdown asciidoc insists to wrap everything.
Cc: Danilo Cesar Lemes de Paula danilo.cesar@collabora.co.uk Cc: Thomas Wood thomas.wood@intel.com Cc: Jonathan Corbet corbet@lwn.net Signed-off-by: Daniel Vetter daniel.vetter@intel.com --- Documentation/DocBook/Makefile | 6 +++--- scripts/kernel-doc | 18 ++++++++++++++---- 2 files changed, 17 insertions(+), 7 deletions(-)
diff --git a/Documentation/DocBook/Makefile b/Documentation/DocBook/Makefile index 246ad38550e5..5335955c0de5 100644 --- a/Documentation/DocBook/Makefile +++ b/Documentation/DocBook/Makefile @@ -104,8 +104,8 @@ define rule_docproc endef
%.xml: %.tmpl $(KERNELDOC) $(DOCPROC) $(KERNELDOCXMLREF) FORCE - @(which pandoc > /dev/null 2>&1) || \ - (echo "*** To get propper documentation you need to install pandoc ***";) + @(which asciidoc > /dev/null 2>&1) || \ + (echo "*** To get propper documentation you need to install asciidoc ***";) $(call if_changed_rule,docproc)
# Tell kbuild to always build the programs @@ -116,7 +116,7 @@ notfoundtemplate = echo "*** You have to install docbook-utils or xmlto ***"; \ db2xtemplate = db2TYPE -o $(dir $@) $< xmltotemplate = xmlto TYPE $(XMLTOFLAGS) -o $(dir $@) $<
-ifneq ($(shell which pandoc >/dev/null 2>&1 && echo found),found) +ifneq ($(shell which asciidoc >/dev/null 2>&1 && echo found),found) MARKDOWNREADY := ""; endif
diff --git a/scripts/kernel-doc b/scripts/kernel-doc index e01e74f15a22..cbfa4c03189e 100755 --- a/scripts/kernel-doc +++ b/scripts/kernel-doc @@ -524,7 +524,7 @@ sub dump_doc_section { sub markdown_to_docbook { my $orig_content = $_[0];
- my $pid = open3( *CHLD_IN, *CHLD_OUT, *CHLD_ERR, "pandoc --columns=80 -f markdown -t docbook" ); + my $pid = open3( *CHLD_IN, *CHLD_OUT, *CHLD_ERR, "asciidoc -a 'newline=\n' --no-header-footer --backend=docbook45 -" );
print CHLD_IN "$orig_content"; close(CHLD_IN); @@ -540,9 +540,9 @@ sub markdown_to_docbook { close(CHLD_ERR);
if ($output_markdown_nopara) { - # pandoc insists in adding Main <para></para>, sometimes we - # want to remove them. - $content =~ s:\A\s*<para>\s*\n(.*)\n</para>\Z$:$1:egsm; + # asciidoc insists in adding Main <simpara></simpara>, sometimes + # we want to remove them. + $content =~ s:\A\s*<simpara>(.*)</simpara>\Z:$1:egsm; }
return $content; @@ -605,6 +605,16 @@ sub output_highlight { # print STDERR "contents af:$contents\n"; if ($use_markdown) { $contents = markdown_to_docbook($contents); + + # Compared to markdown asciidoc doesn't let through arbitrary xml + # markup. We need to un-escape the kerneldoc markup for functions, + # structures, ... + $contents =~ s/<quote>(\S*)</quote>/<quote>$1</quote>/g; + $contents =~ s/<constant>(\S*)</constant>/<constant>$1</constant>/g; + $contents =~ s/<structname>(\S*)</structname>/<structname>$1</structname>/g; + $contents =~ s/<parameter>(\S*)</parameter>/<parameter>$1</parameter>/g; + $contents =~ s/<function>(\S*)</function>/<function>$1</function>/g; + $contents =~ s/<envar>(\S*)</envar>/<envar>$1</envar>/g; }
# strip whitespaces when generating html5
On Wed, 02 Dec 2015, Daniel Vetter daniel.vetter@ffwll.ch wrote:
By popular demand.
Unpopular if you ask me.
Also, I think what you're trying to say is, use asciidoc the tool instead of pandoc the tool, and as a side effect change the markup. Or even, use a tool written in one language instead of a tool written in another, and as a side effect change the markup.
In the end, I guess I would just like to have the technical and factual reasons for this change in the commit message, instead of referring to some mythical unverifiable popular demand.
BR, Jani.
This needs some adjustment/fixups after feeding snippets to asciidoc since compared to markdown asciidown escapes xml markup and doesn't just let it through.
The other noticeable change is that build times increase a lot - we need to launch the markup process per-snippet, there's a few thousand of them and asciidoc (python) has a substantial higher overhead per invocation than pandoc (haskell).
v2: More fine-tuning:
Use unixe newlines, not the default dos ones. Only results in ugliness in the intermediate gpu.xml, but still.
Resurrect the hack to remove paragraphs for the one-line references. Like markdown asciidoc insists to wrap everything.
Cc: Danilo Cesar Lemes de Paula danilo.cesar@collabora.co.uk Cc: Thomas Wood thomas.wood@intel.com Cc: Jonathan Corbet corbet@lwn.net Signed-off-by: Daniel Vetter daniel.vetter@intel.com
Documentation/DocBook/Makefile | 6 +++--- scripts/kernel-doc | 18 ++++++++++++++---- 2 files changed, 17 insertions(+), 7 deletions(-)
diff --git a/Documentation/DocBook/Makefile b/Documentation/DocBook/Makefile index 246ad38550e5..5335955c0de5 100644 --- a/Documentation/DocBook/Makefile +++ b/Documentation/DocBook/Makefile @@ -104,8 +104,8 @@ define rule_docproc endef
%.xml: %.tmpl $(KERNELDOC) $(DOCPROC) $(KERNELDOCXMLREF) FORCE
- @(which pandoc > /dev/null 2>&1) || \
- (echo "*** To get propper documentation you need to install pandoc ***";)
- @(which asciidoc > /dev/null 2>&1) || \
- (echo "*** To get propper documentation you need to install asciidoc ***";) $(call if_changed_rule,docproc)
# Tell kbuild to always build the programs @@ -116,7 +116,7 @@ notfoundtemplate = echo "*** You have to install docbook-utils or xmlto ***"; \ db2xtemplate = db2TYPE -o $(dir $@) $< xmltotemplate = xmlto TYPE $(XMLTOFLAGS) -o $(dir $@) $<
-ifneq ($(shell which pandoc >/dev/null 2>&1 && echo found),found) +ifneq ($(shell which asciidoc >/dev/null 2>&1 && echo found),found) MARKDOWNREADY := ""; endif
diff --git a/scripts/kernel-doc b/scripts/kernel-doc index e01e74f15a22..cbfa4c03189e 100755 --- a/scripts/kernel-doc +++ b/scripts/kernel-doc @@ -524,7 +524,7 @@ sub dump_doc_section { sub markdown_to_docbook { my $orig_content = $_[0];
- my $pid = open3( *CHLD_IN, *CHLD_OUT, *CHLD_ERR, "pandoc --columns=80 -f markdown -t docbook" );
my $pid = open3( *CHLD_IN, *CHLD_OUT, *CHLD_ERR, "asciidoc -a 'newline=\n' --no-header-footer --backend=docbook45 -" );
print CHLD_IN "$orig_content"; close(CHLD_IN);
@@ -540,9 +540,9 @@ sub markdown_to_docbook { close(CHLD_ERR);
if ($output_markdown_nopara) {
# pandoc insists in adding Main <para></para>, sometimes we# want to remove them.$content =~ s:\A\s*<para>\s*\n(.*)\n</para>\Z$:$1:egsm;
# asciidoc insists in adding Main <simpara></simpara>, sometimes# we want to remove them.$content =~ s:\A\s*<simpara>(.*)</simpara>\Z:$1:egsm;}
return $content;
@@ -605,6 +605,16 @@ sub output_highlight { # print STDERR "contents af:$contents\n"; if ($use_markdown) { $contents = markdown_to_docbook($contents);
- # Compared to markdown asciidoc doesn't let through arbitrary xml
- # markup. We need to un-escape the kerneldoc markup for functions,
- # structures, ...
- $contents =~ s/<quote>(\S*)</quote>/<quote>$1</quote>/g;
- $contents =~ s/<constant>(\S*)</constant>/<constant>$1</constant>/g;
- $contents =~ s/<structname>(\S*)</structname>/<structname>$1</structname>/g;
- $contents =~ s/<parameter>(\S*)</parameter>/<parameter>$1</parameter>/g;
- $contents =~ s/<function>(\S*)</function>/<function>$1</function>/g;
- $contents =~ s/<envar>(\S*)</envar>/<envar>$1</envar>/g; }
# strip whitespaces when generating html5
On Wed, Dec 02, 2015 at 03:33:43PM +0200, Jani Nikula wrote:
On Wed, 02 Dec 2015, Daniel Vetter daniel.vetter@ffwll.ch wrote:
By popular demand.
Unpopular if you ask me.
Also, I think what you're trying to say is, use asciidoc the tool instead of pandoc the tool, and as a side effect change the markup. Or even, use a tool written in one language instead of a tool written in another, and as a side effect change the markup.
In the end, I guess I would just like to have the technical and factual reasons for this change in the commit message, instead of referring to some mythical unverifiable popular demand.
When discussing with Dave at KS whether we could do some kind of markup for drm only he asked me to use asciidoc instead of pandoc. So popular = 100% of the relevant maintainer (which is just Dave).
Suprisingly the markup doesn't really change much since mostly asciidoc is just a (massive) extension of the markdown markup we've used. The changes are really just in how it's glued into kernel-doc. At least I didn't notice anything else yet. So yep, it's really just a tooling exchange that resulted in markup language changes.
What I probably should mention is that asciidoc has better table support, which is something we've run into a bit with markdown already (our property tables ain't pretty). -Daniel
BR, Jani.
This needs some adjustment/fixups after feeding snippets to asciidoc since compared to markdown asciidown escapes xml markup and doesn't just let it through.
The other noticeable change is that build times increase a lot - we need to launch the markup process per-snippet, there's a few thousand of them and asciidoc (python) has a substantial higher overhead per invocation than pandoc (haskell).
v2: More fine-tuning:
Use unixe newlines, not the default dos ones. Only results in ugliness in the intermediate gpu.xml, but still.
Resurrect the hack to remove paragraphs for the one-line references. Like markdown asciidoc insists to wrap everything.
Cc: Danilo Cesar Lemes de Paula danilo.cesar@collabora.co.uk Cc: Thomas Wood thomas.wood@intel.com Cc: Jonathan Corbet corbet@lwn.net Signed-off-by: Daniel Vetter daniel.vetter@intel.com
Documentation/DocBook/Makefile | 6 +++--- scripts/kernel-doc | 18 ++++++++++++++---- 2 files changed, 17 insertions(+), 7 deletions(-)
diff --git a/Documentation/DocBook/Makefile b/Documentation/DocBook/Makefile index 246ad38550e5..5335955c0de5 100644 --- a/Documentation/DocBook/Makefile +++ b/Documentation/DocBook/Makefile @@ -104,8 +104,8 @@ define rule_docproc endef
%.xml: %.tmpl $(KERNELDOC) $(DOCPROC) $(KERNELDOCXMLREF) FORCE
- @(which pandoc > /dev/null 2>&1) || \
- (echo "*** To get propper documentation you need to install pandoc ***";)
- @(which asciidoc > /dev/null 2>&1) || \
- (echo "*** To get propper documentation you need to install asciidoc ***";) $(call if_changed_rule,docproc)
# Tell kbuild to always build the programs @@ -116,7 +116,7 @@ notfoundtemplate = echo "*** You have to install docbook-utils or xmlto ***"; \ db2xtemplate = db2TYPE -o $(dir $@) $< xmltotemplate = xmlto TYPE $(XMLTOFLAGS) -o $(dir $@) $<
-ifneq ($(shell which pandoc >/dev/null 2>&1 && echo found),found) +ifneq ($(shell which asciidoc >/dev/null 2>&1 && echo found),found) MARKDOWNREADY := ""; endif
diff --git a/scripts/kernel-doc b/scripts/kernel-doc index e01e74f15a22..cbfa4c03189e 100755 --- a/scripts/kernel-doc +++ b/scripts/kernel-doc @@ -524,7 +524,7 @@ sub dump_doc_section { sub markdown_to_docbook { my $orig_content = $_[0];
- my $pid = open3( *CHLD_IN, *CHLD_OUT, *CHLD_ERR, "pandoc --columns=80 -f markdown -t docbook" );
my $pid = open3( *CHLD_IN, *CHLD_OUT, *CHLD_ERR, "asciidoc -a 'newline=\n' --no-header-footer --backend=docbook45 -" );
print CHLD_IN "$orig_content"; close(CHLD_IN);
@@ -540,9 +540,9 @@ sub markdown_to_docbook { close(CHLD_ERR);
if ($output_markdown_nopara) {
# pandoc insists in adding Main <para></para>, sometimes we# want to remove them.$content =~ s:\A\s*<para>\s*\n(.*)\n</para>\Z$:$1:egsm;
# asciidoc insists in adding Main <simpara></simpara>, sometimes# we want to remove them.$content =~ s:\A\s*<simpara>(.*)</simpara>\Z:$1:egsm;}
return $content;
@@ -605,6 +605,16 @@ sub output_highlight { # print STDERR "contents af:$contents\n"; if ($use_markdown) { $contents = markdown_to_docbook($contents);
- # Compared to markdown asciidoc doesn't let through arbitrary xml
- # markup. We need to un-escape the kerneldoc markup for functions,
- # structures, ...
- $contents =~ s/<quote>(\S*)</quote>/<quote>$1</quote>/g;
- $contents =~ s/<constant>(\S*)</constant>/<constant>$1</constant>/g;
- $contents =~ s/<structname>(\S*)</structname>/<structname>$1</structname>/g;
- $contents =~ s/<parameter>(\S*)</parameter>/<parameter>$1</parameter>/g;
- $contents =~ s/<function>(\S*)</function>/<function>$1</function>/g;
- $contents =~ s/<envar>(\S*)</envar>/<envar>$1</envar>/g; }
# strip whitespaces when generating html5
-- Jani Nikula, Intel Open Source Technology Center
On Wed, 02 Dec 2015, Daniel Vetter daniel@ffwll.ch wrote:
On Wed, Dec 02, 2015 at 03:33:43PM +0200, Jani Nikula wrote:
On Wed, 02 Dec 2015, Daniel Vetter daniel.vetter@ffwll.ch wrote:
By popular demand.
Unpopular if you ask me.
Also, I think what you're trying to say is, use asciidoc the tool instead of pandoc the tool, and as a side effect change the markup. Or even, use a tool written in one language instead of a tool written in another, and as a side effect change the markup.
In the end, I guess I would just like to have the technical and factual reasons for this change in the commit message, instead of referring to some mythical unverifiable popular demand.
When discussing with Dave at KS whether we could do some kind of markup for drm only he asked me to use asciidoc instead of pandoc. So popular = 100% of the relevant maintainer (which is just Dave).
Please put that in the commit message then.
Suprisingly the markup doesn't really change much since mostly asciidoc is just a (massive) extension of the markdown markup we've used. The changes are really just in how it's glued into kernel-doc. At least I didn't notice anything else yet. So yep, it's really just a tooling exchange that resulted in markup language changes.
What I probably should mention is that asciidoc has better table support, which is something we've run into a bit with markdown already (our property tables ain't pretty).
I'll live.
BR, Jani.
-Daniel
BR, Jani.
This needs some adjustment/fixups after feeding snippets to asciidoc since compared to markdown asciidown escapes xml markup and doesn't just let it through.
The other noticeable change is that build times increase a lot - we need to launch the markup process per-snippet, there's a few thousand of them and asciidoc (python) has a substantial higher overhead per invocation than pandoc (haskell).
v2: More fine-tuning:
Use unixe newlines, not the default dos ones. Only results in ugliness in the intermediate gpu.xml, but still.
Resurrect the hack to remove paragraphs for the one-line references. Like markdown asciidoc insists to wrap everything.
Cc: Danilo Cesar Lemes de Paula danilo.cesar@collabora.co.uk Cc: Thomas Wood thomas.wood@intel.com Cc: Jonathan Corbet corbet@lwn.net Signed-off-by: Daniel Vetter daniel.vetter@intel.com
Documentation/DocBook/Makefile | 6 +++--- scripts/kernel-doc | 18 ++++++++++++++---- 2 files changed, 17 insertions(+), 7 deletions(-)
diff --git a/Documentation/DocBook/Makefile b/Documentation/DocBook/Makefile index 246ad38550e5..5335955c0de5 100644 --- a/Documentation/DocBook/Makefile +++ b/Documentation/DocBook/Makefile @@ -104,8 +104,8 @@ define rule_docproc endef
%.xml: %.tmpl $(KERNELDOC) $(DOCPROC) $(KERNELDOCXMLREF) FORCE
- @(which pandoc > /dev/null 2>&1) || \
- (echo "*** To get propper documentation you need to install pandoc ***";)
- @(which asciidoc > /dev/null 2>&1) || \
- (echo "*** To get propper documentation you need to install asciidoc ***";) $(call if_changed_rule,docproc)
# Tell kbuild to always build the programs @@ -116,7 +116,7 @@ notfoundtemplate = echo "*** You have to install docbook-utils or xmlto ***"; \ db2xtemplate = db2TYPE -o $(dir $@) $< xmltotemplate = xmlto TYPE $(XMLTOFLAGS) -o $(dir $@) $<
-ifneq ($(shell which pandoc >/dev/null 2>&1 && echo found),found) +ifneq ($(shell which asciidoc >/dev/null 2>&1 && echo found),found) MARKDOWNREADY := ""; endif
diff --git a/scripts/kernel-doc b/scripts/kernel-doc index e01e74f15a22..cbfa4c03189e 100755 --- a/scripts/kernel-doc +++ b/scripts/kernel-doc @@ -524,7 +524,7 @@ sub dump_doc_section { sub markdown_to_docbook { my $orig_content = $_[0];
- my $pid = open3( *CHLD_IN, *CHLD_OUT, *CHLD_ERR, "pandoc --columns=80 -f markdown -t docbook" );
my $pid = open3( *CHLD_IN, *CHLD_OUT, *CHLD_ERR, "asciidoc -a 'newline=\n' --no-header-footer --backend=docbook45 -" );
print CHLD_IN "$orig_content"; close(CHLD_IN);
@@ -540,9 +540,9 @@ sub markdown_to_docbook { close(CHLD_ERR);
if ($output_markdown_nopara) {
# pandoc insists in adding Main <para></para>, sometimes we# want to remove them.$content =~ s:\A\s*<para>\s*\n(.*)\n</para>\Z$:$1:egsm;
# asciidoc insists in adding Main <simpara></simpara>, sometimes# we want to remove them.$content =~ s:\A\s*<simpara>(.*)</simpara>\Z:$1:egsm;}
return $content;
@@ -605,6 +605,16 @@ sub output_highlight { # print STDERR "contents af:$contents\n"; if ($use_markdown) { $contents = markdown_to_docbook($contents);
- # Compared to markdown asciidoc doesn't let through arbitrary xml
- # markup. We need to un-escape the kerneldoc markup for functions,
- # structures, ...
- $contents =~ s/<quote>(\S*)</quote>/<quote>$1</quote>/g;
- $contents =~ s/<constant>(\S*)</constant>/<constant>$1</constant>/g;
- $contents =~ s/<structname>(\S*)</structname>/<structname>$1</structname>/g;
- $contents =~ s/<parameter>(\S*)</parameter>/<parameter>$1</parameter>/g;
- $contents =~ s/<function>(\S*)</function>/<function>$1</function>/g;
- $contents =~ s/<envar>(\S*)</envar>/<envar>$1</envar>/g; }
# strip whitespaces when generating html5
-- Jani Nikula, Intel Open Source Technology Center
Unfortunately the entire improved docbook project died at KS in a massive bikeshed. So we need to carry this around in drm private trees forever :(
I discussed this with Dave at kernel summit and he's ok with this.
I'll maintain the asciidoc support in topic/kerneldoc in the drm-intel.git tree. There's an autobuilder that pushes drm-intel-nightly (which includes all of Dave's trees) to
http://dri.freedesktop.org/docs/drm/
Thomas Wood keeps care of that autobuilder, so please ping him if there's something amiss.
Note that asciidoc is optional - all the kerneldoc comment layout rules are the same, those bits landed in 4.4. It will simply not quite look as pretty if you don't have it installed.
Cc: Dave Airlie airlied@redhat.com Cc: Thomas Wood thomas.wood@intel.com Cc: Jonathan Corbet corbet@lwn.net Acked-by: Dave Airlie airlied@redhat.com Signed-off-by: Daniel Vetter daniel.vetter@intel.com --- Documentation/DocBook/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Documentation/DocBook/Makefile b/Documentation/DocBook/Makefile index 5335955c0de5..b655343631cf 100644 --- a/Documentation/DocBook/Makefile +++ b/Documentation/DocBook/Makefile @@ -17,7 +17,7 @@ DOCBOOKS := z8530book.xml device-drivers.xml \ tracepoint.xml gpu.xml media_api.xml w1.xml \ writing_musb_glue_layer.xml crypto-API.xml iio.xml
-MARKDOWNREADY := +MARKDOWNREADY := gpu.xml
include Documentation/DocBook/media/Makefile
On Wed, 25 Nov 2015 18:07:59 +0100 Daniel Vetter daniel.vetter@ffwll.ch wrote:
Unfortunately the entire improved docbook project died at KS in a massive bikeshed. So we need to carry this around in drm private trees forever :(
I don't think that's an entirely helpful way to look at things, honestly. Changing how the docs system works affects a lot of people, they're going to have input on the matter.
And I sure hope it hasn't "died". The posting of these patches suggests that perhaps it has not.
Anyway, I wanted to say that, my silence notwithstanding, I haven't just dropped these. I had some hassles to deal with (replacing the entire LWN server infrastructure, for example) but those are done; I hope to be able to mess with this stuff a bit in the very near future.
Have the table-rendering issues that Graham talked about before gotten any better with the newer scheme?
Thanks,
jon
On Fri, Dec 11, 2015 at 03:12:26PM -0700, Jonathan Corbet wrote:
On Wed, 25 Nov 2015 18:07:59 +0100 Daniel Vetter daniel.vetter@ffwll.ch wrote:
Unfortunately the entire improved docbook project died at KS in a massive bikeshed. So we need to carry this around in drm private trees forever :(
I don't think that's an entirely helpful way to look at things, honestly. Changing how the docs system works affects a lot of people, they're going to have input on the matter.
And I sure hope it hasn't "died". The posting of these patches suggests that perhaps it has not.
Hm, I think I fumbled the cc for the cover letter:
http://www.spinics.net/lists/intel-gfx/msg81351.html
In short it hasn't died, and 4.5 will have a few thousand more lines of doc already employing asciidoc (and ofc also the other stuff that already landed in 4.4). I just figured there's no way this could get it, and I'd much rather improve the docs themselves than trying to convince core kernel folks that this might be useful.
Anyway, I wanted to say that, my silence notwithstanding, I haven't just dropped these. I had some hassles to deal with (replacing the entire LWN server infrastructure, for example) but those are done; I hope to be able to mess with this stuff a bit in the very near future.
Have the table-rendering issues that Graham talked about before gotten any better with the newer scheme?
I haven't tackled the table conversion story yet, but one reason for me to switch to asciidoc (besides that Dave thought it was a good idea) was the better table support. I haven't tried, but it /should/ be powerful enough. Another bonus is in-line figures as asciiart that get rendered to png. But haven't done the integration work on that yet either.
Anyway things seem to work and I'm happy.
Thanks, Daniel
On Sat, 12 Dec 2015 12:13:45 +0100 Daniel Vetter daniel@ffwll.ch wrote:
I just figured there's no way this could get it, and I'd much rather improve the docs themselves than trying to convince core kernel folks that this might be useful.
So I'm not quite sure why you figured that; I never said it, certainly.
I've been messing with it a bit, seems to work. I do still wish we could consider alternatives, especially those that might simplify the toolchain rather than complicating it. But it's clear that I'm not succeeding in finding time to actually explore that idea; the contents of $EXCUSES are good, but the end result is the same. And the patch fairy just isn't coming through for me on this one.
In my mind, there's clearly no good that can come from (further) delaying something that works in favor of an "it would be nice" that may never even exist. So I'm currently thinking that I'll pull this into the docs tree once the merge window is done, with the plan to push it for 4.6. Then we can see if anybody screams.
That gives a couple of weeks for an updated patch set, should you have one.
The build-time increase is painful in the extreme - about a factor of three for a -j1 build, and that's with only one file using the feature. It feels wrong, somehow, for the docs build to take longer than building the kernel itself. Can we do something about that?
- How many of the comments actually use asciidoc features? Might there be some possibility of detecting those in kernel-doc and skipping the callout to asciidoc when it's not needed?
- Pandoc seems to do asciidoc. I still don't like the idea of depending on it for this to work, but having the *option* to use it is fine. If it's really that much faster (yes, Python startup is painful) then maybe providing the option is worth it.
- All over the kernel we've seen that batching improves performance. It would take a bit of work, but I bet kernel-doc could put together all the snippets from one file, pass them through a single asciidoc invocation, then split the results back apart. That would probably eliminate the performance hit entirely.
None of that is a condition for pulling this stuff in, but can it be looked into?
Thanks,
jon
On Tue, 12 Jan 2016, Jonathan Corbet corbet@lwn.net wrote:
In my mind, there's clearly no good that can come from (further) delaying something that works in favor of an "it would be nice" that may never even exist. So I'm currently thinking that I'll pull this into the docs tree once the merge window is done, with the plan to push it for 4.6. Then we can see if anybody screams.
Must... resist... urge to bikeshed about the choice of markup...
The build-time increase is painful in the extreme - about a factor of three for a -j1 build, and that's with only one file using the feature. It feels wrong, somehow, for the docs build to take longer than building the kernel itself. Can we do something about that?
"Holy big-O, batman. Asciidoc appears to be quadractically slow." [1]
Fortunately the same quote lead me to asciidoctor [2], which was maybe twice as fast as asciidoc. An improvement, but could be much better.
BR, Jani.
[1] https://twitter.com/marijnjh/status/473935469676216321 [2] http://asciidoctor.org/docs/asciidoc-asciidoctor-diffs/
On Mon, Jan 11, 2016 at 06:12:12PM -0700, Jonathan Corbet wrote:
On Sat, 12 Dec 2015 12:13:45 +0100 Daniel Vetter daniel@ffwll.ch wrote:
I just figured there's no way this could get it, and I'd much rather improve the docs themselves than trying to convince core kernel folks that this might be useful.
So I'm not quite sure why you figured that; I never said it, certainly.
To clarify this wasn't really my impression of your stance, but of the overall room opinion when we had the discussion at KS. And then my main goal here is to write great docs for drm (we have about 3k lines more docs in 4.5 already), so that's why I dropped the ball on upstreaming. It seemed unlikely to succeed, at least without some really seriuos effort at convincing everyone, all while the drm docs for atomic haven't been in good shape yet. Since then we had a few contributors of new atomic drivers note on irc already that "oh cool, this is documented now". Overall really just boils down to what I see as the most important things for drm ;-)
I've been messing with it a bit, seems to work. I do still wish we could consider alternatives, especially those that might simplify the toolchain rather than complicating it. But it's clear that I'm not succeeding in finding time to actually explore that idea; the contents of $EXCUSES are good, but the end result is the same. And the patch fairy just isn't coming through for me on this one.
In my mind, there's clearly no good that can come from (further) delaying something that works in favor of an "it would be nice" that may never even exist. So I'm currently thinking that I'll pull this into the docs tree once the merge window is done, with the plan to push it for 4.6. Then we can see if anybody screams.
That gives a couple of weeks for an updated patch set, should you have one.
The build-time increase is painful in the extreme - about a factor of three for a -j1 build, and that's with only one file using the feature. It feels wrong, somehow, for the docs build to take longer than building the kernel itself. Can we do something about that?
- How many of the comments actually use asciidoc features? Might there be some possibility of detecting those in kernel-doc and skipping the callout to asciidoc when it's not needed?
I think that amounts to writing a partial parser (we use asciidoc for tables, lists, links, formatting, code snippets by now already, someone even thought of using the asciiart->png feature it has but it's not yet wired up). I don't think it's feasible.
- Pandoc seems to do asciidoc. I still don't like the idea of depending on it for this to work, but having the *option* to use it is fine. If it's really that much faster (yes, Python startup is painful) then maybe providing the option is worth it.
Hm, Dave asked me to convert to use python-based asciidoc insted of haskell-based pandoc.
- All over the kernel we've seen that batching improves performance. It would take a bit of work, but I bet kernel-doc could put together all the snippets from one file, pass them through a single asciidoc invocation, then split the results back apart. That would probably eliminate the performance hit entirely.
None of that is a condition for pulling this stuff in, but can it be looked into?
Besides what Jani mention that asciidoctor should be a drop-in replacement if installed it also seems possible to parallelize the call-out to kernel-doc from docproc.c without too much effort. I hoped Jani would get around to implement the asciidoctor support, and I'm hoping I can snipe away some free sometimes the next few months to look at docproc.c more seriously. This would kinda be a cool intern project, but atm we throw them all at improving testing infrastructure ...
Anyway I'm of course still open to get this upstream, and I think a few things should be polished (like the speed-up). But right now bandwidth on my side isn't too plentiful. Maybe we should aim to have a few better ideas (perhaps even for all of the docs stuff) for next KS and respin that discussion?
Thanks, Daniel
On Tue, 2016-01-12 at 09:34 +0100, Daniel Vetter wrote:
On Mon, Jan 11, 2016 at 06:12:12PM -0700, Jonathan Corbet wrote:
On Sat, 12 Dec 2015 12:13:45 +0100 Daniel Vetter daniel@ffwll.ch wrote:
I just figured there's no way this could get it, and I'd much rather improve the docs themselves than trying to convince core kernel folks that this might be useful.
So I'm not quite sure why you figured that; I never said it, certainly.
To clarify this wasn't really my impression of your stance, but of the overall room opinion when we had the discussion at KS. And then my main goal here is to write great docs for drm (we have about 3k lines more docs in 4.5 already), so that's why I dropped the ball on upstreaming. It seemed unlikely to succeed, at least without some really seriuos effort at convincing everyone, all while the drm docs for atomic haven't been in good shape yet. Since then we had a few contributors of new atomic drivers note on irc already that "oh cool, this is documented now". Overall really just boils down to what I see as the most important things for drm ; -)
I've been messing with it a bit, seems to work. I do still wish we could consider alternatives, especially those that might simplify the toolchain rather than complicating it. But it's clear that I'm not succeeding in finding time to actually explore that idea; the contents of $EXCUSES are good, but the end result is the same. And the patch fairy just isn't coming through for me on this one.
In my mind, there's clearly no good that can come from (further) delaying something that works in favor of an "it would be nice" that may never even exist. So I'm currently thinking that I'll pull this into the docs tree once the merge window is done, with the plan to push it for 4.6. Then we can see if anybody screams.
That gives a couple of weeks for an updated patch set, should you have one.
The build-time increase is painful in the extreme - about a factor of three for a -j1 build, and that's with only one file using the feature. It feels wrong, somehow, for the docs build to take longer than building the kernel itself. Can we do something about that?
- How many of the comments actually use asciidoc features? Might
there be some possibility of detecting those in kernel-doc and skipping the callout to asciidoc when it's not needed?
I think that amounts to writing a partial parser (we use asciidoc for tables, lists, links, formatting, code snippets by now already, someone even thought of using the asciiart->png feature it has but it's not yet wired up). I don't think it's feasible.
- Pandoc seems to do asciidoc. I still don't like the idea of
depending on it for this to work, but having the *option* to use it is fine. If it's really that much faster (yes, Python startup is painful) then maybe providing the option is worth it.
Hm, Dave asked me to convert to use python-based asciidoc insted of haskell-based pandoc.
- All over the kernel we've seen that batching improves
performance. It would take a bit of work, but I bet kernel-doc could put together all the snippets from one file, pass them through a single asciidoc invocation, then split the results back apart. That would probably eliminate the performance hit entirely.
None of that is a condition for pulling this stuff in, but can it be looked into?
Besides what Jani mention that asciidoctor should be a drop-in replacement if installed it also seems possible to parallelize the call-out to kernel-doc from docproc.c without too much effort. I hoped Jani would get around to implement the asciidoctor support, and I'm hoping I can snipe away some free sometimes the next few months to look at docproc.c more seriously. This would kinda be a cool intern project, but atm we throw them all at improving testing infrastructure ...
Anyway I'm of course still open to get this upstream, and I think a few things should be polished (like the speed-up). But right now bandwidth on my side isn't too plentiful. Maybe we should aim to have a few better ideas (perhaps even for all of the docs stuff) for next KS and respin that discussion?
I was just about to reply to the thread.... looking at the linux.conf.au schedule it would seem that you are both attending and presenting, and there appears to be some sort of Documentation mini -summit on the Monday as well (not sure if that is the place for a discussion though). I will be at LCA for the Wed-Fri. You may not have to wait until the next KS?
Graham
Thanks, Daniel
On Tue, Jan 12, 2016 at 11:06:17AM +0000, Graham Whaley wrote:
On Tue, 2016-01-12 at 09:34 +0100, Daniel Vetter wrote:
On Mon, Jan 11, 2016 at 06:12:12PM -0700, Jonathan Corbet wrote:
On Sat, 12 Dec 2015 12:13:45 +0100 Daniel Vetter daniel@ffwll.ch wrote:
I just figured there's no way this could get it, and I'd much rather improve the docs themselves than trying to convince core kernel folks that this might be useful.
So I'm not quite sure why you figured that; I never said it, certainly.
To clarify this wasn't really my impression of your stance, but of the overall room opinion when we had the discussion at KS. And then my main goal here is to write great docs for drm (we have about 3k lines more docs in 4.5 already), so that's why I dropped the ball on upstreaming. It seemed unlikely to succeed, at least without some really seriuos effort at convincing everyone, all while the drm docs for atomic haven't been in good shape yet. Since then we had a few contributors of new atomic drivers note on irc already that "oh cool, this is documented now". Overall really just boils down to what I see as the most important things for drm ; -)
I've been messing with it a bit, seems to work. I do still wish we could consider alternatives, especially those that might simplify the toolchain rather than complicating it. But it's clear that I'm not succeeding in finding time to actually explore that idea; the contents of $EXCUSES are good, but the end result is the same. And the patch fairy just isn't coming through for me on this one.
In my mind, there's clearly no good that can come from (further) delaying something that works in favor of an "it would be nice" that may never even exist. So I'm currently thinking that I'll pull this into the docs tree once the merge window is done, with the plan to push it for 4.6. Then we can see if anybody screams.
That gives a couple of weeks for an updated patch set, should you have one.
The build-time increase is painful in the extreme - about a factor of three for a -j1 build, and that's with only one file using the feature. It feels wrong, somehow, for the docs build to take longer than building the kernel itself. Can we do something about that?
- How many of the comments actually use asciidoc features? Might
there be some possibility of detecting those in kernel-doc and skipping the callout to asciidoc when it's not needed?
I think that amounts to writing a partial parser (we use asciidoc for tables, lists, links, formatting, code snippets by now already, someone even thought of using the asciiart->png feature it has but it's not yet wired up). I don't think it's feasible.
- Pandoc seems to do asciidoc. I still don't like the idea of
depending on it for this to work, but having the *option* to use it is fine. If it's really that much faster (yes, Python startup is painful) then maybe providing the option is worth it.
Hm, Dave asked me to convert to use python-based asciidoc insted of haskell-based pandoc.
- All over the kernel we've seen that batching improves
performance. It would take a bit of work, but I bet kernel-doc could put together all the snippets from one file, pass them through a single asciidoc invocation, then split the results back apart. That would probably eliminate the performance hit entirely.
None of that is a condition for pulling this stuff in, but can it be looked into?
Besides what Jani mention that asciidoctor should be a drop-in replacement if installed it also seems possible to parallelize the call-out to kernel-doc from docproc.c without too much effort. I hoped Jani would get around to implement the asciidoctor support, and I'm hoping I can snipe away some free sometimes the next few months to look at docproc.c more seriously. This would kinda be a cool intern project, but atm we throw them all at improving testing infrastructure ...
Anyway I'm of course still open to get this upstream, and I think a few things should be polished (like the speed-up). But right now bandwidth on my side isn't too plentiful. Maybe we should aim to have a few better ideas (perhaps even for all of the docs stuff) for next KS and respin that discussion?
I was just about to reply to the thread.... looking at the linux.conf.au schedule it would seem that you are both attending and presenting, and there appears to be some sort of Documentation mini -summit on the Monday as well (not sure if that is the place for a discussion though). I will be at LCA for the Wed-Fri. You may not have to wait until the next KS?
Sounds like a great idea to pick this up at lca and toss around for some. -Daniel
On Tue, 12 Jan 2016, Jonathan Corbet corbet@lwn.net wrote:
On Sat, 12 Dec 2015 12:13:45 +0100 Daniel Vetter daniel@ffwll.ch wrote:
I just figured there's no way this could get it, and I'd much rather improve the docs themselves than trying to convince core kernel folks that this might be useful.
So I'm not quite sure why you figured that; I never said it, certainly.
I've been messing with it a bit, seems to work. I do still wish we could consider alternatives, especially those that might simplify the toolchain rather than complicating it. But it's clear that I'm not succeeding in finding time to actually explore that idea; the contents of $EXCUSES are good, but the end result is the same. And the patch fairy just isn't coming through for me on this one.
In my mind, there's clearly no good that can come from (further) delaying something that works in favor of an "it would be nice" that may never even exist. So I'm currently thinking that I'll pull this into the docs tree once the merge window is done, with the plan to push it for 4.6. Then we can see if anybody screams.
That gives a couple of weeks for an updated patch set, should you have one.
The build-time increase is painful in the extreme - about a factor of three for a -j1 build, and that's with only one file using the feature. It feels wrong, somehow, for the docs build to take longer than building the kernel itself. Can we do something about that?
How many of the comments actually use asciidoc features? Might there be some possibility of detecting those in kernel-doc and skipping the callout to asciidoc when it's not needed?
Pandoc seems to do asciidoc. I still don't like the idea of depending on it for this to work, but having the *option* to use it is fine. If it's really that much faster (yes, Python startup is painful) then maybe providing the option is worth it.
All over the kernel we've seen that batching improves performance. It would take a bit of work, but I bet kernel-doc could put together all the snippets from one file, pass them through a single asciidoc invocation, then split the results back apart. That would probably eliminate the performance hit entirely.
I had another look at the whole thing, inspired by your lwn article [1].
I am guessing the whole design of calling the markup tool (asciidoc or pandoc or whatever) from kernel-doc for every documentation comment comes from trying really hard to minimize changes to the rest of the documentation system. From trying not to touch the DocBook parts so much. And thus the documentation build works really hard to convert thousands of markup snippets to DocBook and then again back to a bunch of other formats like html or pdf.
What if we added support for some markup language as an alternative to DocBook for the high level documentation? What if we taught kernel-doc to output said markup natively, and included those generated pieces into the high level documentation using the markup's own include directives? At least AsciiDoc and reStructuredText support this.
kernel-doc already supports several output formats, DocBook, HTML, plain text, man, whatnot. It should not be a big deal to add another. Even though I don't do perl (and that's the main reason I've generally steered clear anything kernel-doc) I toyed with adding reStructuredText support, and I got some sensible output surprisingly quickly.
This would mean *one* kernel-doc invocation per source file included, and *one* markup tool invocation per high level document. Additionally there would have to be dependency generation from the high level document to the included files.
As a bonus, one could write (and *read*!) the high level documentation in a markup that's meant for humans. If needed, at least AsciiDoc supports generating DocBook, which I suppose could be fed back to the existing documentation pipeline.
None of that is a condition for pulling this stuff in, but can it be looked into?
To be clear, I also don't want this idea to block the code that we have now from being merged. This is a longer term thing anyway. But I'd like to explore the alternatives.
Thoughts?
BR, Jani.
[1] https://lwn.net/Articles/671496/
On Thu, 14 Jan 2016 22:03:26 +0200 Jani Nikula jani.nikula@linux.intel.com wrote:
What if we added support for some markup language as an alternative to DocBook for the high level documentation? What if we taught kernel-doc to output said markup natively, and included those generated pieces into the high level documentation using the markup's own include directives? At least AsciiDoc and reStructuredText support this.
That is kind of what I've been thinking. I think we could dispense with docbook entirely, simplifying the toolchain and making the template files easier to read and edit. Something like that would also make it easy to keep the regular .txt documents in a structured form and to integrate them into the "formatted" documents if it makes sense.
Jani, want to join us at LCA and talk about it? :)
jon
On Thu, 14 Jan 2016, Jonathan Corbet corbet@lwn.net wrote:
On Thu, 14 Jan 2016 22:03:26 +0200 Jani Nikula jani.nikula@linux.intel.com wrote:
What if we added support for some markup language as an alternative to DocBook for the high level documentation? What if we taught kernel-doc to output said markup natively, and included those generated pieces into the high level documentation using the markup's own include directives? At least AsciiDoc and reStructuredText support this.
That is kind of what I've been thinking. I think we could dispense with docbook entirely, simplifying the toolchain and making the template files easier to read and edit. Something like that would also make it easy to keep the regular .txt documents in a structured form and to integrate them into the "formatted" documents if it makes sense.
Thanks, this is enough encouragement that maybe I'll toy around with this a little more in my, uh, copious free time.
Jani, want to join us at LCA and talk about it? :)
Heh, sure I'd *want* to...
BR, Jani.
dri-devel@lists.freedesktop.org
-
Daniel Vetter -
Daniel Vetter -
Graham Whaley -
Jani Nikula -
Jonathan Corbet