How can I update a .yml file, ignoring preexisting Jinja syntax, using Python?
I have some preprocessing to do with some existing .yml files - however, some of them have Jinja template syntax embedded in them:
A:
B:
- ip: 1.2.3.4
- myArray:
- {{ jinja.variable }}
- val1
- val2
I'd want to read in this file, and add val3 under myArray as such:
A:
B:
- ip: 1.2.3.4
- myArray:
- {{ jinja.variable }}
- val1
- val2
- val 3
I tried manually writing out the jinja templates, but they got written with single quotes around them: '{{ jinja.variable }}'
What's the recommended way for me to read such .yml files and modify them, albeit with preexisting Jinja syntax? I'd like to add information to these files keeping all else the same.
I tried the above using PyYAML on Python 2.7+
The solution in this answer has been incorporated into ruamel.yaml using a plugin mechanism. At the bottom of this post there are quick-and-dirty instructions on how to use that.
There are three aspects in updating a YAML file that contains jinja2 "code":
- making the jinja2 code acceptable to the YAML parser
- making sure the acceptable can reversed (i.e. the changes should be unique, so only they get reversed)
- preserving the layout of the YAML file so that the updated file processed by jinja2 still produces a valid YAML file, that again can be loaded.
Let's start by making your example somewhat more realistic by adding a jinja2 variable definition and for-loop and adding some comments (input.yaml):
# trying to update
{% set xyz = "123" }
A:
B:
- ip: 1.2.3.4
- myArray:
- {{ jinja.variable }}
- val1
- val2 # add a value after this one
{% for d in data %}
- phone: {{ d.phone }}
name: {{ d.name }}
{% endfor %}
- {{ xyz }}
# #% or ##% should not be in the file and neither <{ or <<{
The lines starting with {% contain no YAML, so we'll make those into comments (assuming that comments are preserved on round-trip, see below). Since YAML scalars cannot start with { without being quoted we'll change the {{ to <{. This is done in the following code by calling sanitize() (which also stores the patterns used, and the reverse is done in sanitize.reverse (using the stored patterns).
The preservation of your YAML code (block-style etc) is best done using ruamel.yaml (disclaimer: I am the author of that package), that way you don't have to worry about flow-style elements in the input getting mangled into as block style as with the rather crude default_flow_style=False that the other answers use. ruamel.yaml also preserves comments, both the ones that were originally in the file, as well as those temporarily inserted to "comment out" jinja2 constructs starting with %{.
The resulting code:
import sys
from ruamel.yaml import YAML
yaml = YAML()
class Sanitize:
"""analyse, change and revert YAML/jinja2 mixture to/from valid YAML"""
def __init__(self):
self.accacc = None
self.accper = None
def __call__(self, s):
len = 1
for len in range(1, 10):
pat = '<' * len + '{'
if pat not in s:
self.accacc = pat
break
else:
raise NotImplementedError('could not find substitute pattern '+pat)
len = 1
for len in range(1, 10):
pat = '#' * len + '%'
if pat not in s:
self.accper = pat
break
else:
raise NotImplementedError('could not find substitute pattern '+pat)
return s.replace('{{', self.accacc).replace('{%', self.accper)
def revert(self, s):
return s.replace(self.accacc, '{{').replace(self.accper, '{%')
def update_one(file_name, out_file_name=None):
sanitize = Sanitize()
with open(file_name) as fp:
data = yaml.load(sanitize(fp.read()))
myArray = data['A']['B'][1]['myArray']
pos = myArray.index('val2')
myArray.insert(pos+1, 'val 3')
if out_file_name is None:
yaml.dump(data, sys.stdout, transform=sanitize.revert)
else:
with open(out_file_name, 'w') as fp:
yaml.dump(data, out, transform=sanitize.revert)
update_one('input.yaml')
which prints (specify a second parameter to update_one() to write to a file) using Python 2.7:
# trying to update
{% set xyz = "123" }
A:
B:
- ip: 1.2.3.4
- myArray:
- {{ jinja.variable }}
- val1
- val2 # add a value after this one
- val 3
{% for d in data %}
- phone: {{ d.phone }}
name: {{ d.name }}
{% endfor %}
- {{ xyz }}
# #% or ##% should not be in the file and neither <{ or <<{
If neither #{ nor <{ are in any of the original inputs then sanitizing and reverting can be done with simple one-line functions (see this versions of this post), and then you don't need the class Sanitize
Your example is indented with one position (key B) as well as two positions (the sequence elements), ruamel.yaml doesn't have that fine control over output indentation (and I don't know of any YAML parser that does). The indent (defaulting to 2) is applied to both YAML mappings as to sequence elements (measured to the beginning of the element, not to the dash). This has no influence on re-reading the YAML and happened to the output of the other two answerers as well (without them pointing out this change).
Also note that YAML().load() is safe (i.e. doesn't load arbitrary potentially malicious objects), whereas the yaml.load() as used in the other answers is definitely unsafe, it says so in the documentation and is even mentioned in the WikiPedia article on YAML. If you use yaml.load(), you would have to check each and every input file to make sure there are no tagged objects that could cause your disc to be wiped (or worse).
If you need to update your files repeatedly, and have control over the jinja2 templating, it might be better to change the patterns for jinja2 once and not revert them, and then specifying appropriate block_start_string, variable_start_string (and possible block_end_string and variable_end_string) to the jinja2.FileSystemLoader added as loader to the jinja2.Environment.
If the above seems to complicated then in a a virtualenv do:
pip install ruamel.yaml ruamel.yaml.jinja2
assuming you have the input.yaml from before you can run:
import os
from ruamel.yaml import YAML
yaml = YAML(typ='jinja2')
with open('input.yaml') as fp:
data = yaml.load(fp)
myArray = data['A']['B'][1]['myArray']
pos = myArray.index('val2')
myArray.insert(pos+1, 'val 3')
with open('output.yaml', 'w') as fp:
yaml.dump(data, fp)
os.system('diff -u input.yaml output.yaml')
to get the diff output:
--- input.yaml 2017-06-14 23:10:46.144710495 +0200
+++ output.yaml 2017-06-14 23:11:21.627742055 +0200
@@ -8,6 +8,7 @@
- {{ jinja.variable }}
- val1
- val2 # add a value after this one
+ - val 3
{% for d in data %}
- phone: {{ d.phone }}
name: {{ d.name }}
ruamel.yaml 0.15.7 implements a new plug-in mechanism and ruamel.yaml.jinja2 is a plug-in that rewraps the code in this answer transparently for the user. Currently the information for reversion is attached to the YAML() instance, so make sure you do yaml = YAML(typ='jinja2') for each file you process (that information could be attached to the top-level data instance, just like the YAML comments are).
One way to do this is to use the jinja2 parser itself to parse the template and output an alternate format.
Jinja2 Code:
This code inherits from the Jinja2 Parser, Lexer and Environment classes to parse inside variable blocks (usually {{ }}). Instead of evaluating the variables, this code changes the text to something that yaml can understand. The exact same code can be used to reverse the process with an exchange of the delimiters. By default it translates to the delimiters suggested by snakecharmerb.
import jinja2
import yaml
class MyParser(jinja2.parser.Parser):
def parse_tuple(self, *args, **kwargs):
super(MyParser, self).parse_tuple(*args, **kwargs)
if not isinstance(self.environment._jinja_vars, list):
node_text = self.environment._jinja_vars
self.environment._jinja_vars = None
return jinja2.nodes.Const(
self.environment.new_variable_start_string +
node_text +
self.environment.new_variable_end_string)
class MyLexer(jinja2.lexer.Lexer):
def __init__(self, *args, **kwargs):
super(MyLexer, self).__init__(*args, **kwargs)
self.environment = None
def tokenize(self, source, name=None, filename=None, state=None):
stream = self.tokeniter(source, name, filename, state)
def my_stream(environment):
for t in stream:
if environment._jinja_vars is None:
if t[1] == 'variable_begin':
self.environment._jinja_vars = []
elif t[1] == 'variable_end':
node_text = ''.join(
[x[2] for x in self.environment._jinja_vars])
self.environment._jinja_vars = node_text
else:
environment._jinja_vars.append(t)
yield t
return jinja2.lexer.TokenStream(self.wrap(
my_stream(self.environment), name, filename), name, filename)
jinja2.lexer.Lexer = MyLexer
class MyEnvironment(jinja2.Environment):
def __init__(self,
new_variable_start_string='<<',
new_variable_end_string='>>',
reverse=False,
*args,
**kwargs):
if kwargs.get('loader') is None:
kwargs['loader'] = jinja2.BaseLoader()
super(MyEnvironment, self).__init__(*args, **kwargs)
self._jinja_vars = None
if reverse:
self.new_variable_start_string = self.variable_start_string
self.new_variable_end_string = self.variable_end_string
self.variable_start_string = new_variable_start_string
self.variable_end_string = new_variable_end_string
else:
self.new_variable_start_string = new_variable_start_string
self.new_variable_end_string = new_variable_end_string
self.lexer.environment = self
def _parse(self, source, name, filename):
return MyParser(self, source, name,
jinja2._compat.encode_filename(filename)).parse()
How/Why?
The jinja2 parser scans the template file looking for delimiters. When finding delimiters, it then switches to parse the appropriate material between the delimiters. The changes in the code here insert themselves into the lexer and parser to capture the text captured during the template compilation, and then when finding the termination delimiter, concats the parsed tokens into a string and inserts it as a jinja2.nodes.Const parse node, in place of the compiled jinja code, so that when the template is rendered the string is inserted instead of a variable expansion.
The MyEnvironment() code is used to hook in the custom parser and lexer extensions. And while at it, added some parameters processing.
The primary advantage of this approach is that it should be fairly robust to parsing whatever jinja will parse.
User Code:
def dict_from_yaml_template(template_string):
env = MyEnvironment()
template = env.from_string(template_string)
return yaml.load(template.render())
def yaml_template_from_dict(template_yaml, **kwargs):
env = MyEnvironment(reverse=True)
template = env.from_string(yaml.dump(template_yaml, **kwargs))
return template.render()
Test Code:
with open('data.yml') as f:
data = dict_from_yaml_template(f.read())
data['A']['B'][1]['myArray'].append('val 3')
data['A']['B'][1]['myArray'].append('<< jinja.variable2 >>')
new_yaml = yaml_template_from_dict(data, default_flow_style=False)
print(new_yaml)
data.yml
A:
B:
- ip: 1.2.3.4
- myArray:
- {{ x['}}'] }}
- {{ [(1, 2, (3, 4))] }}
- {{ jinja.variable }}
- val1
- val2
Results:
A:
B:
- ip: 1.2.3.4
- myArray:
- {{ x['}}'] }}
- {{ [(1, 2, (3, 4))] }}
- {{ jinja.variable }}
- val1
- val2
- val 3
- {{ jinja.variable2 }}