From: Markus Heiser markus.heiser@darmarit.de

This patch brings scalable figure, image handling and a concept to embed *render* markups:

* DOT (http://www.graphviz.org) * SVG

For image handling use the 'image' replacement::

.. kernel-image:: svg_image.svg :alt: simple SVG image

For figure handling use the 'figure' replacement::

.. kernel-figure:: svg_image.svg :alt: simple SVG image

SVG image example

Embed *render* markups (or languages) like Graphviz's **DOT** is provided by the *render* directive.::

.. kernel-render:: DOT :alt: foobar digraph :caption: Embedded **DOT** (Graphviz) code.

digraph foo { "bar" -> "baz"; }

The *render* directive is a concept to integrate *render* markups and languages, yet supported markups:

* DOT: render embedded Graphviz's **DOT** * SVG: render embedded Scalable Vector Graphics (**SVG**)

v2: s/DOC/DOT/ in a few places (by Daniel).

v3: Simplify stuff a bit (by Daniel):

- Remove path detection and setup/check code for that. In Documentation/media/Makefile we already simply use these tools, better to have one consolidated check if we want/need one. Also remove the convertsvg support, we require ImageMagick's convert already in the doc build, no need for a 2nd fallback.

- Use sphinx for depency tracking, remove hand-rolled version.

- Forward stderr from dot and convert, otherwise debugging issues with the diagrams is impossible.

v4: Only sphinx 1.4 (released in Mar 2016) has patches.Figure. Implement Markus suggestion for backwards compatability with earlier releases. Laurent reported this, running sphinx 1.3. Solution entirely untested.

v5: Use an explicit version check (suggested by Laurent).

Cc: Jonathan Corbet corbet@lwn.net Cc: linux-doc@vger.kernel.org Cc: Jani Nikula jani.nikula@linux.intel.com Cc: Mauro Carvalho Chehab mchehab@s-opensource.com Cc: Markus Heiser markus.heiser@darmarit.de Cc: Laurent Pinchart laurent.pinchart@ideasonboard.com Acked-by: Markus Heiser markus.heiser@darmarit.de Tested-by: Laurent Pinchart laurent.pinchart@ideasonboard.com Signed-off-by: Markus Heiser markus.heiser@darmarit.de (v1) Signed-off-by: Daniel Vetter daniel.vetter@intel.com --- Documentation/conf.py | 2 +- Documentation/doc-guide/hello.dot | 3 + Documentation/doc-guide/sphinx.rst | 90 ++++++- Documentation/doc-guide/svg_image.svg | 10 + Documentation/process/changes.rst | 7 +- Documentation/sphinx/kfigure.py | 450 ++++++++++++++++++++++++++++++++++ 6 files changed, 556 insertions(+), 6 deletions(-) create mode 100644 Documentation/doc-guide/hello.dot create mode 100644 Documentation/doc-guide/svg_image.svg create mode 100644 Documentation/sphinx/kfigure.py

diff --git a/Documentation/conf.py b/Documentation/conf.py index f6823cf01275..e3f537ce2935 100644 --- a/Documentation/conf.py +++ b/Documentation/conf.py @@ -34,7 +34,7 @@ from load_config import loadConfig # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. -extensions = ['kerneldoc', 'rstFlatTable', 'kernel_include', 'cdomain'] +extensions = ['kerneldoc', 'rstFlatTable', 'kernel_include', 'cdomain', 'kfigure']

# The name of the math extension changed on Sphinx 1.4 if major == 1 and minor > 3: diff --git a/Documentation/doc-guide/hello.dot b/Documentation/doc-guide/hello.dot new file mode 100644 index 000000000000..504621dfc595 --- /dev/null +++ b/Documentation/doc-guide/hello.dot @@ -0,0 +1,3 @@ +graph G { + Hello -- World +} diff --git a/Documentation/doc-guide/sphinx.rst b/Documentation/doc-guide/sphinx.rst index 532d65b70500..b902744ce7dd 100644 --- a/Documentation/doc-guide/sphinx.rst +++ b/Documentation/doc-guide/sphinx.rst @@ -34,8 +34,10 @@ format-specific subdirectories under ``Documentation/output``.

To generate documentation, Sphinx (``sphinx-build``) must obviously be installed. For prettier HTML output, the Read the Docs Sphinx theme -(``sphinx_rtd_theme``) is used if available. For PDF output, ``rst2pdf`` is also -needed. All of these are widely available and packaged in distributions. +(``sphinx_rtd_theme``) is used if available. For PDF output you'll also need +``XeLaTeX`` and CairoSVG (http://cairosvg.org) or alternatively ``convert(1)`` +from ImageMagick (https://www.imagemagick.org). All of these are widely +available and packaged in distributions.

To pass extra options to Sphinx, you can use the ``SPHINXOPTS`` make variable. For example, use ``make SPHINXOPTS=-v htmldocs`` to get more verbose @@ -232,3 +234,87 @@ Rendered as: * .. _`last row`:

- column 3 + + +Figures & Images +================ + +If you want to add an image, you should use the ``kernel-figure`` and +``kernel-image`` directives. E.g. to insert a figure with a scalable +image format use SVG:: + + .. kernel-figure:: svg_image.svg + :alt: simple SVG image + + SVG image example + +.. kernel-figure:: svg_image.svg + :alt: simple SVG image + + SVG image example + +The kernel figure (and image) directive support **DOT** formated files, see + +* DOT: http://graphviz.org/pdf/dotguide.pdf +* Graphviz: http://www.graphviz.org/content/dot-language + +A simple example:: + + .. kernel-figure:: hello.dot + :alt: hello world + + DOT's hello world example + +.. kernel-figure:: hello.dot + :alt: hello world + + DOT's hello world example + +Embed *render* markups (or languages) like Graphviz's **DOT** is provided by the +``kernel-render`` directives.:: + + .. kernel-render:: DOT + :alt: foobar digraph + :caption: Embedded **DOT** (Graphviz) code. + + digraph foo { + "bar" -> "baz"; + } + +How this will be rendered depends on the installed tools. If Graphviz is +installed, you will see an vector image. If not the raw markup is inserted as +*literal-block*. + +.. kernel-render:: DOT + :alt: foobar digraph + :caption: Embedded **DOT** (Graphviz) code. + + digraph foo { + "bar" -> "baz"; + } + +The *render* directive has all the options known from the *figure* directive, +plus option ``caption``. If ``caption`` has a value, a *figure* node is +inserted. If not, a *image* node is inserted. + +Embedded **SVG**:: + + .. kernel-render:: SVG + :caption: Embedded **SVG** markup. + :alt: so-nw-arrow + + <?xml version="1.0" encoding="UTF-8"?> + <svg xmlns="http://www.w3.org/2000/svg" version="1.1" ...> + ... + </svg> + +.. kernel-render:: SVG + :caption: Embedded **SVG** markup. + :alt: so-nw-arrow + + <?xml version="1.0" encoding="UTF-8"?> + <svg xmlns="http://www.w3.org/2000/svg" + version="1.1" baseProfile="full" width="70px" height="40px" viewBox="0 0 700 400"> + <line x1="180" y1="370" x2="500" y2="50" stroke="black" stroke-width="15px"/> + <polygon points="585 0 525 25 585 50" transform="rotate(135 525 25)"/> + </svg> diff --git a/Documentation/doc-guide/svg_image.svg b/Documentation/doc-guide/svg_image.svg new file mode 100644 index 000000000000..5405f85b8137 --- /dev/null +++ b/Documentation/doc-guide/svg_image.svg @@ -0,0 +1,10 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- originate: https://commons.wikimedia.org/wiki/File:Variable_Resistor.svg --> +<svg xmlns="http://www.w3.org/2000/svg" + version="1.1" baseProfile="full" + width="70px" height="40px" viewBox="0 0 700 400"> + <line x1="0" y1="200" x2="700" y2="200" stroke="black" stroke-width="20px"/> + <rect x="100" y="100" width="500" height="200" fill="white" stroke="black" stroke-width="20px"/> + <line x1="180" y1="370" x2="500" y2="50" stroke="black" stroke-width="15px"/> + <polygon points="585 0 525 25 585 50" transform="rotate(135 525 25)"/> +</svg> diff --git a/Documentation/process/changes.rst b/Documentation/process/changes.rst index 56ce66114665..e4f25038ef65 100644 --- a/Documentation/process/changes.rst +++ b/Documentation/process/changes.rst @@ -318,9 +318,10 @@ PDF outputs, it is recommended to use version 1.4.6. .. note::

Please notice that, for PDF and LaTeX output, you'll also need ``XeLaTeX`` - version 3.14159265. Depending on the distribution, you may also need - to install a series of ``texlive`` packages that provide the minimal - set of functionalities required for ``XeLaTex`` to work. + version 3.14159265. Depending on the distribution, you may also need to + install a series of ``texlive`` packages that provide the minimal set of + functionalities required for ``XeLaTex`` to work. For PDF output you'll also + need ``convert(1)`` from ImageMagick (https://www.imagemagick.org).

Other tools ----------- diff --git a/Documentation/sphinx/kfigure.py b/Documentation/sphinx/kfigure.py new file mode 100644 index 000000000000..32eab0f4cfba --- /dev/null +++ b/Documentation/sphinx/kfigure.py @@ -0,0 +1,450 @@ +# -*- coding: utf-8; mode: python -*- +# pylint: disable=C0103 +u""" + scalable figure and image handling + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + Sphinx extension which implements scalable image handling. + + :copyright: Copyright (C) 2016 Markus Heiser + :license: GPL Version 2, June 1991 see Linux/COPYING for details. + + The build for image formats depence on image's source format and output's + destination format. This extension implement methods to simplify image + handling from the author's POV. Directives like ``kernel-figure`` implement + methods *to* always get the best output-format even if some tools are not + installed.For more details take a look at ``convert_image(...)`` which is + the core of all conversions. + + * ``.. kernel-image``: for image handling / ``.. image::`` replacement + + * ``.. kernel-figure``: for figure handling / ``.. figure::`` replacement + + * ``.. kernel-render``: for render markup / a concept to embed *render* + markups (or languages). Supported markups (see ``RENDER_MARKUP_EXT``) + + + ``DOT``: render embedded Graphviz's **DOC** + + ``SVG``: render embedded Scalable Vector Graphics (**SVG**) + + ... *developable* + + Used tools: + + * ``dot(1)``: Graphviz (http://www.graphviz.org). If Graphviz is not + available, the DOT language is inserted as literal-block. + + * SVG to PDF: To generate PDF, you need at least one of this tools: + + - ``convert(1)``: ImageMagick (https://www.imagemagick.org) + + List of customizations: + + * generate PDF from SVG / used by PDF (LaTeX) builder + + * generate SVG (html-builder) and PDF (latex-builder) from DOT files. + DOT: see http://www.graphviz.org/content/dot-language + + """ + +import os +from os import path +import subprocess +from hashlib import sha1 +import sys + +from docutils import nodes +from docutils.statemachine import ViewList +from docutils.parsers.rst import directives +from docutils.parsers.rst.directives import images + +import sphinx + +# Get Sphinx version +major, minor, patch = map(int, sphinx.__version__.split(".")) +if major == 1 and minor > 3: + # patches.Figure only landed in Sphinx 1.4 + from sphinx.directives.patches import Figure +else: + Figure = images.Figure + +__version__ = '1.0' + +# simple helper +# ------------- + +def mkdir(folder, mode=0o775): + if not path.isdir(folder): + os.makedirs(folder, mode) + +# def debug_handle(self, node): # pylint: disable=W0613 +# from linuxdoc.kernel_doc import CONSOLE +# CONSOLE() + +def pass_handle(self, node): # pylint: disable=W0613 + pass + +# setup conversion tools and sphinx extension +# ------------------------------------------- + +def setup(app): + # image handling + app.add_directive("kernel-image", KernelImage) + app.add_node(kernel_image, + html = (visit_kernel_image, pass_handle), + latex = (visit_kernel_image, pass_handle), + texinfo = (visit_kernel_image, pass_handle), + text = (visit_kernel_image, pass_handle), + man = (visit_kernel_image, pass_handle), ) + + # figure handling + app.add_directive("kernel-figure", KernelFigure) + app.add_node(kernel_figure, + html = (visit_kernel_figure, pass_handle), + latex = (visit_kernel_figure, pass_handle), + texinfo = (visit_kernel_figure, pass_handle), + text = (visit_kernel_figure, pass_handle), + man = (visit_kernel_figure, pass_handle), ) + + # render handling + app.add_directive('kernel-render', KernelRender) + app.add_node(kernel_render, + html = (visit_kernel_render, pass_handle), + latex = (visit_kernel_render, pass_handle), + texinfo = (visit_kernel_render, pass_handle), + text = (visit_kernel_render, pass_handle), + man = (visit_kernel_render, pass_handle), ) + + return dict( + version = __version__, + parallel_read_safe = True, + parallel_write_safe = True + ) + +# integrate conversion tools +# -------------------------- + +RENDER_MARKUP_EXT = { + # The '.ext' must be handled by convert_image(..) function's *in_ext* input. + # <name> : <.ext> + 'DOT' : '.dot' + , 'SVG' : '.svg' +} + +def convert_image(img_node, translator): # pylint: disable=R0912 + """Convert an image node for the builder. + + Different builder prefer different image formats, e.g. *latex* builder + prefer PDF while *html* builder prefer SVG format for images. + + This function handles outputs image formats in depence of source the format + of the image and the translator's output format. This also means to + manipulate/update the *image* dictionary of the builder (``builder.images``) + + """ + fname, in_ext = path.splitext(path.basename(img_node['uri'])) + src_fname = path.join(translator.builder.srcdir, img_node['uri']) + src_folder = path.dirname(img_node['uri']) + out_dir = translator.builder.outdir + dst_fname = None + + # in kernel builds, use 'make SPHINXOPTS=-v' to see verbose messages + verbose = translator.builder.app.verbose + warn = translator.builder.warn + + verbose('assert best format for: ' + img_node['uri']) + + if in_ext == '.dot': + + # ---------- + # handle DOT + # ---------- + + dst_fname = path.join(out_dir, fname + '.pdf') + + if translator.builder.format == 'html': + dst_fname = path.join(out_dir, src_folder, fname + '.svg') + else: + # all other builder formats will include DOT as raw + with open(src_fname, "r") as dot: + data = dot.read() + node = nodes.literal_block(data, data) + img_node.replace_self(node) + + + elif in_ext == '.svg': + + # ---------- + # handle SVG + # ---------- + + if translator.builder.format == 'latex': + dst_fname = path.join(out_dir, fname + '.pdf') + + if dst_fname: + name = dst_fname[len(out_dir) + 1:] + # the builder needs not to copy one more time, so pop it if exists. + translator.builder.images.pop(img_node['uri'], None) + img_node['uri'] = dst_fname + img_node['candidates'] = {'*': dst_fname} + + mkdir(path.dirname(dst_fname)) + + if in_ext == '.dot': + verbose('convert DOT to: {out}/' + name) + dot2format(src_fname, dst_fname) + + elif in_ext == '.svg': + verbose('convert SVG to: {out}/' + name) + svg2pdf(src_fname, dst_fname) + +def dot2format(dot_fname, out_fname): + """Converts DOT file to ``out_fname`` using ``dot(1)``. + + * ``dot_fname`` pathname of the input DOT file, including extension ``.dot`` + * ``out_fname`` pathname of the output file, including format extension + + The *format extension* depends on the ``dot`` command (see ``man dot`` + option ``-Txxx``). Normally you will use one of the following extensions: + + - ``.ps`` for PostScript, + - ``.svg`` or ``svgz`` for Structured Vector Graphics, + - ``.fig`` for XFIG graphics and + - ``.png`` or ``gif`` for common bitmap graphics. + + """ + out_format = path.splitext(out_fname)[1][1:] + cmd = ['dot', '-T%s' % out_format, dot_fname] + exit_code = 42 + with open(out_fname, "w") as out: + p = subprocess.Popen( + cmd, stdout = out, stderr = subprocess.PIPE ) + nil, err = p.communicate() + + sys.stderr.write(err) + + exit_code = p.returncode + out.flush() + return bool(exit_code == 0) + +def svg2pdf(svg_fname, pdf_fname): + """Converts SVG to PDF with CairoSVG or ``convert(1)`` command. + + Uses ``convert(1)`` from ImageMagick (https://www.imagemagick.org) for + conversion. Returns ``True`` on success and ``False`` if an error occurred + (e.g. none of the conversion tool is available). + + * ``svg_fname`` pathname of the input SVG file with extension (``.svg``) + * ``pdf_name`` pathname of the output PDF file with extension (``.pdf``) + + """ + cmd = [convert_cmd, svg_fname, pdf_fname] + p = subprocess.Popen( + cmd, stdout = out, stderr = subprocess.PIPE ) + nil, err = p.communicate() + + sys.stderr.write(err) + + exit_code = p.returncode + return bool(exit_code == 0) + + +# image handling +# --------------------- + +def visit_kernel_image(self, node): # pylint: disable=W0613 + """Visitor of the ``kernel_image`` Node. + + Handles the ``image`` child-node with the ``convert_image(...)``. + """ + img_node = node[0] + convert_image(img_node, self) + +class kernel_image(nodes.General, nodes.Element): + """Node for ``kernel-image`` directive.""" + pass + +class KernelImage(images.Image): + u"""KernelImage directive + + Earns everything from ``.. image::`` directive, except *remote URI* and + *glob* pattern. The KernelImage wraps a image node into a + kernel_image node. See ``visit_kernel_image``. + """ + + def run(self): + env = self.state.document.settings.env + + uri = self.arguments[0] + if uri.endswith('.*') or uri.find('://') != -1: + raise self.severe( + 'Error in "%s: %s": glob pattern and remote images are not allowed' + % (self.name, uri)) + + # Tell sphinx of the dependency + env.note_dependency(os.path.abspath(uri)) + + result = images.Image.run(self) + if len(result) == 2 or isinstance(result[0], nodes.system_message): + return result + (image_node,) = result + # wrap image node into a kernel_image node / see visitors + node = kernel_image('', image_node) + return [node] + +# figure handling +# --------------------- + +def visit_kernel_figure(self, node): # pylint: disable=W0613 + """Visitor of the ``kernel_figure`` Node. + + Handles the ``image`` child-node with the ``convert_image(...)``. + """ + img_node = node[0][0] + convert_image(img_node, self) + +class kernel_figure(nodes.General, nodes.Element): + """Node for ``kernel-figure`` directive.""" + +class KernelFigure(Figure): + u"""KernelImage directive + + Earns everything from ``.. figure::`` directive, except *remote URI* and + *glob* pattern. The KernelFigure wraps a figure node into a kernel_figure + node. See ``visit_kernel_figure``. + """ + + def run(self): + env = self.state.document.settings.env + + uri = self.arguments[0] + if uri.endswith('.*') or uri.find('://') != -1: + raise self.severe( + 'Error in "%s: %s":' + ' glob pattern and remote images are not allowed' + % (self.name, uri)) + + # Tell sphinx of the dependency + env.note_dependency(os.path.abspath(uri)) + + result = Figure.run(self) + if len(result) == 2 or isinstance(result[0], nodes.system_message): + return result + (figure_node,) = result + # wrap figure node into a kernel_figure node / see visitors + node = kernel_figure('', figure_node) + return [node] + + +# render handling +# --------------------- + +def visit_kernel_render(self, node): + """Visitor of the ``kernel_render`` Node. + + If rendering tools available, save the markup of the ``literal_block`` child + node into a file and replace the ``literal_block`` node with a new created + ``image`` node, pointing to the saved markup file. Afterwards, handle the + image child-node with the ``convert_image(...)``. + """ + + verbose = self.builder.app.verbose + warn = self.builder.warn + srclang = node.get('srclang') + + verbose('visit kernel-render node lang: "%s"' % (srclang)) + + tmp_ext = RENDER_MARKUP_EXT.get(srclang, None) + if tmp_ext is None: + warn('kernel-render: "%s" unknow / include raw.' % (srclang)) + return + + literal_block = node[0] + code = literal_block.astext() + + if tmp_ext: + hashobj = code.encode('utf-8') # str(node.attributes) + fname = '%s-%s' % (srclang, sha1(hashobj).hexdigest()) + tmp_fname = path.join( + self.builder.outdir, self.builder.imagedir, fname + tmp_ext) + + if not path.isfile(tmp_fname): + mkdir(path.dirname(tmp_fname)) + with open(tmp_fname, "w") as out: + out.write(code) + + image_node = nodes.image(node.rawsource, **node.attributes) + image_node['uri'] = tmp_fname + + literal_block.replace_self(image_node) + convert_image(image_node, self) + + +class kernel_render(nodes.General, nodes.Inline, nodes.Element): + """Node for ``kernel-render`` directive.""" + pass + +class KernelRender(Figure): + u"""KernelRender directive + + Render content by external tool. Has all the options known from the + *figure* directive, plus option ``caption``. If ``caption`` has a + value, a figure node with the *caption* is inserted. If not, a image node is + inserted. + + The KernelRender directive wraps the text of the directive into a + literal_block node and wraps it into a kernel_render node. See + ``visit_kernel_render``. + """ + has_content = True + required_arguments = 1 + optional_arguments = 0 + final_argument_whitespace = False + + # earn options from 'figure' + option_spec = Figure.option_spec.copy() + option_spec['caption'] = directives.unchanged + + def run(self): + return [self.build_node()] + + def build_node(self): + + srclang = self.arguments[0].strip() + if srclang not in RENDER_MARKUP_EXT.keys(): + return [self.state_machine.reporter.warning( + 'Unknow source language "%s", use one of: %s.' % ( + srclang, ",".join(RENDER_MARKUP_EXT.keys())), + line=self.lineno)] + + code = '\n'.join(self.content) + if not code.strip(): + return [self.state_machine.reporter.warning( + 'Ignoring "%s" directive without content.' % ( + self.name), + line=self.lineno)] + + node = kernel_render() + node['alt'] = self.options.get('alt','') + node['srclang'] = srclang + literal_node = nodes.literal_block(code, code) + node += literal_node + + caption = self.options.get('caption') + if caption: + # parse cation's content + parsed = nodes.Element() + self.state.nested_parse( + ViewList([caption], source=''), self.content_offset, parsed) + caption_node = nodes.caption( + parsed[0].rawsource, '', *parsed[0].children) + caption_node.source = parsed[0].source + caption_node.line = parsed[0].line + + figure_node = nodes.figure('', node) + for k,v in self.options.items(): + figure_node[k] = v + figure_node += caption_node + + node = figure_node + + return node +

Hi Mauro,

Tested here with the enclosed patch.

great, big step forward making /media/Makefile smaller ... thanks a lot!!!!

It crashed: Exception occurred: File "/devel/v4l/patchwork/Documentation/sphinx/kfigure.py", line 222, in dot2format sys.stderr.write(err) TypeError: write() argument must be str, not bytes The full traceback has been saved in /tmp/sphinx-err-_1vahbmg.log, if you want to report the issue to the developers. Please also report this if it was a user error, so that a better error message can be provided next time. A bug report can be filed in the tracker at https://github.com/sphinx-doc/sphinx/issues. Thanks! Documentation/Makefile.sphinx:69: recipe for target 'htmldocs' failed make[1]: *** [htmldocs] Error 1 Makefile:1450: recipe for target 'htmldocs' failed make: *** [htmldocs] Error 2

Weird enough, it produced a Documentation/output/media/uapi/v4l/pipeline.svg file.

I guess that the dot command writes something to stderr. This is captured by the extension and printed to stderr ...

+def dot2format(dot_fname, out_fname): ... + exit_code = 42 + with open(out_fname, "w") as out: + p = subprocess.Popen( + cmd, stdout = out, stderr = subprocess.PIPE ) + nil, err = p.communicate() + + sys.stderr.write(err) + + exit_code = p.returncode + out.flush() + return bool(exit_code == 0)

File "/devel/v4l/patchwork/Documentation/sphinx/kfigure.py", line 222, in dot2format sys.stderr.write(err) TypeError: write() argument must be str, not bytes

Do we need this stderr output? For a first test, uncomment the "sys.stderr.write(err)“ in line 222. Or, if we really need the stderr, try:

- sys.stderr.write(err) + sys.stderr.write(str(err))

I this fixes, there is another "sys.stderr.write(err)" in func svg2pdf(..) which should also fixed ….

+def svg2pdf(svg_fname, pdf_fname): ... + cmd = [convert_cmd, svg_fname, pdf_fname] + p = subprocess.Popen( + cmd, stdout = out, stderr = subprocess.PIPE ) + nil, err = p.communicate() + - sys.stderr.write(err) + sys.stderr.write(str(err)) + + exit_code = p.returncode + return bool(exit_code == 0)

Thanks!

-- Markus --

Thanks, Mauro

--

[PATCH] docs-rst: Don't use explicit Makefile rules to build SVG and DOT files

Now that we have an extension to handle images, use it.

Signed-off-by: Mauro Carvalho Chehab mchehab@s-opensource.com

diff --git a/Documentation/media/Makefile b/Documentation/media/Makefile index 32663602ff25..5bd52ea1eff0 100644 --- a/Documentation/media/Makefile +++ b/Documentation/media/Makefile @@ -1,51 +1,6 @@ -# Rules to convert DOT and SVG to Sphinx images

-SRC_DIR=$(srctree)/Documentation/media

-DOTS = \

• uapi/v4l/pipeline.dot \

-IMAGES = \

• typical_media_device.svg \
• uapi/dvb/dvbstb.svg \
• uapi/v4l/bayer.svg \
• uapi/v4l/constraints.svg \
• uapi/v4l/crop.svg \
• uapi/v4l/fieldseq_bt.svg \
• uapi/v4l/fieldseq_tb.svg \
• uapi/v4l/nv12mt.svg \
• uapi/v4l/nv12mt_example.svg \
• uapi/v4l/pipeline.svg \
• uapi/v4l/selection.svg \
• uapi/v4l/subdev-image-processing-full.svg \
• uapi/v4l/subdev-image-processing-scaling-multi-source.svg \
• uapi/v4l/subdev-image-processing-crop.svg \
• uapi/v4l/vbi_525.svg \
• uapi/v4l/vbi_625.svg \
• uapi/v4l/vbi_hsync.svg \

-DOTTGT := $(patsubst %.dot,%.svg,$(DOTS)) -IMGDOT := $(patsubst %,$(SRC_DIR)/%,$(DOTTGT))

-IMGTGT := $(patsubst %.svg,%.pdf,$(IMAGES)) -IMGPDF := $(patsubst %,$(SRC_DIR)/%,$(IMGTGT))

-cmd = $(echo-cmd) $(cmd_$(1))

-quiet_cmd_genpdf = GENPDF $2

•  cmd_genpdf = convert $2 $3
  

-quiet_cmd_gendot = DOT $2

•  cmd_gendot = dot -Tsvg $2 > $3
  

-%.pdf: %.svg

• @$(call cmd,genpdf,$<,$@)

-%.svg: %.dot

• @$(call cmd,gendot,$<,$@)

# Rules to convert a .h file to inline RST documentation

+SRC_DIR=$(srctree)/Documentation/media PARSER = $(srctree)/Documentation/sphinx/parse-headers.pl UAPI = $(srctree)/include/uapi/linux KAPI = $(srctree)/include/linux diff --git a/Documentation/media/intro.rst b/Documentation/media/intro.rst index 8f7490c9a8ef..5562fba1d51d 100644 --- a/Documentation/media/intro.rst +++ b/Documentation/media/intro.rst @@ -13,8 +13,8 @@ A typical media device hardware is shown at :ref:`typical_media_device`.

.. _typical_media_device:

-.. figure:: typical_media_device.*

• :alt: typical_media_device.pdf / typical_media_device.svg

+.. kernel-figure:: typical_media_device.svg

• :alt: typical_media_device.svg :align: center

  Typical Media Device

diff --git a/Documentation/media/uapi/dvb/intro.rst b/Documentation/media/uapi/dvb/intro.rst index 2ed5c23102b4..0e76067a5b58 100644 --- a/Documentation/media/uapi/dvb/intro.rst +++ b/Documentation/media/uapi/dvb/intro.rst @@ -55,8 +55,8 @@ Overview

.. _stb_components:

-.. figure:: dvbstb.*

• :alt: dvbstb.pdf / dvbstb.svg

+.. kernel-figure:: dvbstb.svg

• :alt: dvbstb.svg :align: center

  Components of a DVB card/STB

diff --git a/Documentation/media/uapi/v4l/crop.rst b/Documentation/media/uapi/v4l/crop.rst index be58894c9c89..adea29b14135 100644 --- a/Documentation/media/uapi/v4l/crop.rst +++ b/Documentation/media/uapi/v4l/crop.rst @@ -53,8 +53,8 @@ Cropping Structures

.. _crop-scale:

-.. figure:: crop.*

• :alt: crop.pdf / crop.svg

+.. kernel-figure:: crop.svg

• :alt: crop.svg :align: center

  Image Cropping, Insertion and Scaling

diff --git a/Documentation/media/uapi/v4l/dev-raw-vbi.rst b/Documentation/media/uapi/v4l/dev-raw-vbi.rst index baf5f2483927..f761d3a93783 100644 --- a/Documentation/media/uapi/v4l/dev-raw-vbi.rst +++ b/Documentation/media/uapi/v4l/dev-raw-vbi.rst @@ -221,8 +221,8 @@ and always returns default parameters as :ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` does

.. _vbi-hsync:

-.. figure:: vbi_hsync.*

• :alt: vbi_hsync.pdf / vbi_hsync.svg

+.. kernel-figure:: vbi_hsync.svg

• :alt: vbi_hsync.svg :align: center

  **Figure 4.1. Line synchronization**

@@ -230,8 +230,8 @@ and always returns default parameters as :ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` does

.. _vbi-525:

-.. figure:: vbi_525.*

• :alt: vbi_525.pdf / vbi_525.svg

+.. kernel-figure:: vbi_525.svg

• :alt: vbi_525.svg :align: center

  **Figure 4.2. ITU-R 525 line numbering (M/NTSC and M/PAL)**

@@ -240,8 +240,8 @@ and always returns default parameters as :ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` does

.. _vbi-625:

-.. figure:: vbi_625.*

• :alt: vbi_625.pdf / vbi_625.svg

+.. kernel-figure:: vbi_625.svg

• :alt: vbi_625.svg :align: center

  **Figure 4.3. ITU-R 625 line numbering**

diff --git a/Documentation/media/uapi/v4l/dev-subdev.rst b/Documentation/media/uapi/v4l/dev-subdev.rst index cd2870180208..0a54e4d95013 100644 --- a/Documentation/media/uapi/v4l/dev-subdev.rst +++ b/Documentation/media/uapi/v4l/dev-subdev.rst @@ -99,8 +99,8 @@ the video sensor and the host image processing hardware.

.. _pipeline-scaling:

-.. figure:: pipeline.*

• :alt: pipeline.pdf / pipeline.svg

+.. kernel-figure:: pipeline.dot

• :alt: pipeline.dot :align: center

  Image Format Negotiation on Pipelines

@@ -404,8 +404,8 @@ selection will refer to the sink pad format dimensions instead.

.. _subdev-image-processing-crop:

-.. figure:: subdev-image-processing-crop.*

• :alt: subdev-image-processing-crop.pdf / subdev-image-processing-crop.svg

+.. kernel-figure:: subdev-image-processing-crop.svg

• :alt: subdev-image-processing-crop.svg :align: center

  **Figure 4.5. Image processing in subdevs: simple crop example**

@@ -421,8 +421,8 @@ pad.

.. _subdev-image-processing-scaling-multi-source:

-.. figure:: subdev-image-processing-scaling-multi-source.*

• :alt: subdev-image-processing-scaling-multi-source.pdf / subdev-image-processing-scaling-multi-source.svg

+.. kernel-figure:: subdev-image-processing-scaling-multi-source.svg

• :alt: subdev-image-processing-scaling-multi-source.svg :align: center

  **Figure 4.6. Image processing in subdevs: scaling with multiple sources**

@@ -437,8 +437,8 @@ an area at location specified by the source crop rectangle from it.

.. _subdev-image-processing-full:

-.. figure:: subdev-image-processing-full.*

• :alt: subdev-image-processing-full.pdf / subdev-image-processing-full.svg

+.. kernel-figure:: subdev-image-processing-full.svg

• :alt: subdev-image-processing-full.svg :align: center

  **Figure 4.7. Image processing in subdevs: scaling and composition with multiple sinks and sources**

diff --git a/Documentation/media/uapi/v4l/field-order.rst b/Documentation/media/uapi/v4l/field-order.rst index e05fb1041363..7a1a0ac8e37a 100644 --- a/Documentation/media/uapi/v4l/field-order.rst +++ b/Documentation/media/uapi/v4l/field-order.rst @@ -141,8 +141,8 @@ enum v4l2_field Field Order, Top Field First Transmitted ========================================

-.. figure:: fieldseq_tb.*

• :alt: fieldseq_tb.pdf / fieldseq_tb.svg

+.. kernel-figure:: fieldseq_tb.svg

• :alt: fieldseq_tb.svg :align: center

@@ -151,7 +151,7 @@ Field Order, Top Field First Transmitted Field Order, Bottom Field First Transmitted ===========================================

-.. figure:: fieldseq_bt.*

• :alt: fieldseq_bt.pdf / fieldseq_bt.svg

+.. kernel-figure:: fieldseq_bt.svg

• :alt: fieldseq_bt.svg :align: center

diff --git a/Documentation/media/uapi/v4l/pixfmt-nv12mt.rst b/Documentation/media/uapi/v4l/pixfmt-nv12mt.rst index 32d0c8743460..a4d64edc6200 100644 --- a/Documentation/media/uapi/v4l/pixfmt-nv12mt.rst +++ b/Documentation/media/uapi/v4l/pixfmt-nv12mt.rst @@ -33,8 +33,8 @@ Layout of macroblocks in memory is presented in the following figure.

.. _nv12mt:

-.. figure:: nv12mt.*

• :alt: nv12mt.pdf / nv12mt.svg

+.. kernel-figure:: nv12mt.svg

• :alt: nv12mt.svg :align: center

  V4L2_PIX_FMT_NV12MT macroblock Z shape memory layout

@@ -50,8 +50,8 @@ interleaved. Height of the buffer is aligned to 32.

.. _nv12mt_ex:

-.. figure:: nv12mt_example.*

• :alt: nv12mt_example.pdf / nv12mt_example.svg

+.. kernel-figure:: nv12mt_example.svg

• :alt: nv12mt_example.svg :align: center

  Example V4L2_PIX_FMT_NV12MT memory layout of macroblocks

diff --git a/Documentation/media/uapi/v4l/selection-api-003.rst b/Documentation/media/uapi/v4l/selection-api-003.rst index 21686f93c38f..04e9384e1be8 100644 --- a/Documentation/media/uapi/v4l/selection-api-003.rst +++ b/Documentation/media/uapi/v4l/selection-api-003.rst @@ -7,8 +7,8 @@ Selection targets

.. _sel-targets-capture:

-.. figure:: selection.*

• :alt: selection.pdf / selection.svg

+.. kernel-figure:: selection.svg

• :alt: selection.svg :align: center

  Cropping and composing targets

diff --git a/Documentation/media/uapi/v4l/subdev-formats.rst b/Documentation/media/uapi/v4l/subdev-formats.rst index d6152c907b8b..4fbf2e29e653 100644 --- a/Documentation/media/uapi/v4l/subdev-formats.rst +++ b/Documentation/media/uapi/v4l/subdev-formats.rst @@ -1514,8 +1514,8 @@ be named ``MEDIA_BUS_FMT_SRGGB10_2X8_PADHI_LE``.

.. _bayer-patterns:

-.. figure:: bayer.*

• :alt: bayer.pdf / bayer.svg

+.. figure:: bayer.svg

• :alt: bayer.svg :align: center

  **Figure 4.8 Bayer Patterns**

Em Thu, 2 Mar 2017 16:34:22 -0300 Mauro Carvalho Chehab mchehab@s-opensource.com escreveu:

Em Thu, 2 Mar 2017 20:06:39 +0100 Markus Heiser markus.heiser@darmarit.de escreveu:

...

Hi Mauro,

...

Tested here with the enclosed patch.

great, big step forward making /media/Makefile smaller ... thanks a lot!!!!

...

It crashed: Exception occurred: File "/devel/v4l/patchwork/Documentation/sphinx/kfigure.py", line 222, in dot2format sys.stderr.write(err) TypeError: write() argument must be str, not bytes The full traceback has been saved in /tmp/sphinx-err-_1vahbmg.log, if you want to report the issue to the developers. Please also report this if it was a user error, so that a better error message can be provided next time. A bug report can be filed in the tracker at https://github.com/sphinx-doc/sphinx/issues. Thanks! Documentation/Makefile.sphinx:69: recipe for target 'htmldocs' failed make[1]: *** [htmldocs] Error 1 Makefile:1450: recipe for target 'htmldocs' failed make: *** [htmldocs] Error 2

Weird enough, it produced a Documentation/output/media/uapi/v4l/pipeline.svg file.

I guess that the dot command writes something to stderr. This is captured by the extension and printed to stderr ...

+def dot2format(dot_fname, out_fname): ...

• exit_code = 42
• with open(out_fname, "w") as out:

•    p = subprocess.Popen(
  

•        cmd, stdout = out, stderr = subprocess.PIPE )
  

•    nil, err = p.communicate()
  

•    sys.stderr.write(err)
  

•    exit_code = p.returncode
  

•    out.flush()
  

• return bool(exit_code == 0)

...

File "/devel/v4l/patchwork/Documentation/sphinx/kfigure.py", line 222, in dot2format sys.stderr.write(err) TypeError: write() argument must be str, not bytes

Do we need this stderr output? For a first test, uncomment the "sys.stderr.write(err)“ in line 222. Or, if we really need the stderr, try:

•    sys.stderr.write(err)
  

•    sys.stderr.write(str(err))
  

Yes, this fixed. I actually did:

•    sys.stderr.write(err)
  

•    sys.stderr.write(str(err))
  

•    sys.stderr.write("\n")
  
  

It is now printing: b''

I added the \n print to avoid it to be mixed with the "writing output" prints.

No idea how to make sense from it - but clearly, the error report logic require some care ;-)

I'm not a Python programmer, but googling about the right syntax for p.communicate(), I suspect that the fix should be similar to this code snippet below.

Without the if, this code:

sys.stderr.write("Error:" + repr(p.communicate()[0]) + "\n")

prints: Error:None

So, I guess the code below is ok.

Markus, please check.

Daniel, Feel free to merge it with the original patch if OK.

diff --git a/Documentation/sphinx/kfigure.py b/Documentation/sphinx/kfigure.py index 32eab0f4cfba..b154c5f17752 100644 --- a/Documentation/sphinx/kfigure.py +++ b/Documentation/sphinx/kfigure.py @@ -217,11 +217,12 @@ def dot2format(dot_fname, out_fname): with open(out_fname, "w") as out: p = subprocess.Popen( cmd, stdout = out, stderr = subprocess.PIPE ) - nil, err = p.communicate() - - sys.stderr.write(err)

exit_code = p.returncode + + if exit_code == 0: + sys.stderr.write("Error:" + repr(p.communicate()[0]) + "\n") + out.flush() return bool(exit_code == 0)

@@ -239,11 +240,12 @@ def svg2pdf(svg_fname, pdf_fname): cmd = [convert_cmd, svg_fname, pdf_fname] p = subprocess.Popen( cmd, stdout = out, stderr = subprocess.PIPE ) - nil, err = p.communicate() - - sys.stderr.write(err)

exit_code = p.returncode + + if exit_code == 0: + sys.stderr.write("Error:" + repr(p.communicate()[0]) + "\n") + return bool(exit_code == 0)

Em Thu, 2 Mar 2017 22:16:49 +0100 Markus Heiser markus.heiser@darmarit.de escreveu:

...

Am 02.03.2017 um 20:34 schrieb Mauro Carvalho Chehab mchehab@s-opensource.com:

Em Thu, 2 Mar 2017 20:06:39 +0100 Markus Heiser markus.heiser@darmarit.de escreveu:

...

Hi Mauro,

...

Tested here with the enclosed patch.

great, big step forward making /media/Makefile smaller ... thanks a lot!!!!

...

It crashed: Exception occurred: File "/devel/v4l/patchwork/Documentation/sphinx/kfigure.py", line 222, in dot2format sys.stderr.write(err) TypeError: write() argument must be str, not bytes The full traceback has been saved in /tmp/sphinx-err-_1vahbmg.log, if you want to report the issue to the developers. Please also report this if it was a user error, so that a better error message can be provided next time. A bug report can be filed in the tracker at https://github.com/sphinx-doc/sphinx/issues. Thanks! Documentation/Makefile.sphinx:69: recipe for target 'htmldocs' failed make[1]: *** [htmldocs] Error 1 Makefile:1450: recipe for target 'htmldocs' failed make: *** [htmldocs] Error 2

Weird enough, it produced a Documentation/output/media/uapi/v4l/pipeline.svg file.

I guess that the dot command writes something to stderr. This is captured by the extension and printed to stderr ...

+def dot2format(dot_fname, out_fname): ...

• exit_code = 42
• with open(out_fname, "w") as out:

•    p = subprocess.Popen(
  

•        cmd, stdout = out, stderr = subprocess.PIPE )
  

•    nil, err = p.communicate()
  

•    sys.stderr.write(err)
  

•    exit_code = p.returncode
  

•    out.flush()
  

• return bool(exit_code == 0)

...

File "/devel/v4l/patchwork/Documentation/sphinx/kfigure.py", line 222, in dot2format sys.stderr.write(err) TypeError: write() argument must be str, not bytes

Do we need this stderr output? For a first test, uncomment the "sys.stderr.write(err)“ in line 222. Or, if we really need the stderr, try:

•    sys.stderr.write(err)
  

•    sys.stderr.write(str(err))  
  

Yes, this fixed. I actually did:

•    sys.stderr.write(err)
  

•    sys.stderr.write(str(err))
  

•    sys.stderr.write("\n")
  
  

It is now printing: b''

I added the \n print to avoid it to be mixed with the "writing output" prints. No idea how to make sense from it - but clearly, the error report logic require some care ;-)

Aargh, I’am a idiot ... I guess 'sys.stderr.write(err)‘ is a artefact of my development, simply drop it and the subprocess.PIPE of stderr also.

• with open(out_fname, "w") as out:

•    p = subprocess.Popen(
  

•        cmd, stdout = out, stderr = subprocess.PIPE )
  

•        cmd, stdout = out)
  

•    nil, err = p.communicate()
  

•    sys.stderr.write(err)
  

•    exit_code = p.returncode
  

•    out.flush()
  

• return bool(exit_code == 0)

I can’t test it ATM, but without redirect stderr, the stderr of the parent process is inherited.

https://docs.python.org/3.6/library/subprocess.html#popen-constructor

The Popen.communicate() always returns a tuple (stdout_data, stderr_data) with above the tuple is always (None, None).

https://docs.python.org/3.6/library/subprocess.html#subprocess.Popen.communi...

"""to get anything other than None in the result tuple, you need to give stdout=PIPE and/or stderr=PIPE too."""

Sorry, that I made all this mistakes, but „here“ I have only mail and web, no dev-env and I miss my emacs ;)

If the suggestion above does not work, I have to investigate more time next weekend.

Hmm... I would be more verbose on output the error code, printing from where the error came, e. g. printing a message like:

Error #0 when calling dot for '/devel/v4l/patchwork/Documentation/media/uapi/v4l/pipeline.dot': None

As on this patch:

diff --git a/Documentation/sphinx/kfigure.py b/Documentation/sphinx/kfigure.py index 32eab0f4cfba..a366a89f4f98 100644 --- a/Documentation/sphinx/kfigure.py +++ b/Documentation/sphinx/kfigure.py @@ -217,11 +217,13 @@ def dot2format(dot_fname, out_fname): with open(out_fname, "w") as out: p = subprocess.Popen( cmd, stdout = out, stderr = subprocess.PIPE ) - nil, err = p.communicate() - - sys.stderr.write(err) + err = p.communicate()

exit_code = p.returncode + + if exit_code != 0: + sys.stderr.write("Error #%d when calling dot for '%s': %s\n" % (exit_code, dot_fname, repr(err[0]))) + out.flush() return bool(exit_code == 0)

@@ -239,11 +241,13 @@ def svg2pdf(svg_fname, pdf_fname): cmd = [convert_cmd, svg_fname, pdf_fname] p = subprocess.Popen( cmd, stdout = out, stderr = subprocess.PIPE ) - nil, err = p.communicate() - - sys.stderr.write(err) + err = p.communicate()

exit_code = p.returncode + + if exit_code != 0: + sys.stderr.write("Error #%d when calling convert for '%s': %s\n" % (exit_code, dot_fname, repr(err[0]))) + return bool(exit_code == 0)