Issue 4687 - Add Context Function get_pages_metadata to Test Site

tests/test_site/globals/get_pages_metadata.py

 import re
 from jinja2 import contextfunction
 
 
 @contextfunction
 def get_pages_metadata(context, filters=None):
     if not isinstance(filters, dict) and filters:
         raise TypeError('Filters are not a dictionary')
 
     return_data = []
     for page_name, _format in context['source'].list_pages():
         data, filename = context['source'].read_page(page_name, _format)
         page_data = parse_page_metadata(data, page_name)
 
         if filter_metadata(filters, page_data) is True:
             return_data.append(page_data)
 
     return return_data
 
 
 def parse_page_metadata(data, page):
     page_metadata = {'page': page}
     lines = data.splitlines(True)
     for i, line in enumerate(lines):
         if not re.search(r'^\s*[\w\-]+\s*=', line):
             break
         name, value = line.split('=', 1)
-        value = tuple(value.strip().split(','))

[juliandoucette, 2017/02/27 21:27:31]

1. It's annoying to have single values as arrays
  - e.g. {{ post[title][0] }} vs {{ post[title] }}
2. We don't want to split all values by ","
  - e.g. we may have commas in dates or the description of a page

[Vasily Kuznetsov, 2017/02/28 11:24:09]

Yeah, makes sense. I suppose then not every metadata field should be treated as
a list by default. Perhaps a reasonable default would be to treat them as a
strings.

However, there are fields that probably should be lists, for example the tags.
We could come up with some kind of notation for those. One way to do it would be
to say that if the value starts with a square bracket ("["), then it's a list
and it should be split by comma so that:

    tags = [foo, bar]

will produce the value which is a list: ["foo", "bar"] and things like this:

    date = Wed, Nov 5

will stay a string: "Wed, Nov 5".

Another way to do it is to allow something like this syntax for lists:

    tags = foo
               bar
               baz
    title = My awesome page

This would be a bit harder to parse and would require changes to CMS core to be
compatible, so it wouldn't be my first preference.

We could also adopt a different approach and treat everything as a string but
replace the exact matching with substring matching. Then if we have this in the
page header:

    tags = foo, bar

and we look for pages tagged with "foo", it would be found. What I don't like
about this approach is that a page with this in the header would also be
matched:

    tags = foobar, quux

but "foobar" seems like a different tag. We could probably fix that by requiring
that if not the whole string is matched, the end of the match that is inside of
the string should be at a word boundary or a comma or something like that. It
would be more fiddly, but we can probably end up with something reasonably
fool-proof. In this case all metadata fields would be returned as strings
(nothing would be converted to a list), which might or might not be what you
want.

Do any of these solutions sound acceptable or do you have other ideas?

[Jon Sonesen, 2017/02/28 11:59:18]

could we not just check that the value is not more than one and if so don't make
it a list?

+        value = value.strip()
+        if value.startswith('[') and value.endswith(']'):
+            value = [element.strip() for element in value[1:-1].split(',')]
         page_metadata[name.strip()] = value
     return page_metadata
 
 
 def filter_metadata(filters, metadata):
     if filters is None:
         return True
     for filter_name, filter_value in filters.items():
         if filter_name not in metadata:
             return False
-        for option in filter_value.split(','):
-            if option not in metadata[filter_name]:

[Vasily Kuznetsov, 2017/01/20 11:53:38]

Here if a list of options was given for the same field name, we require that all
of them are present. This makes sense in some cases but I wonder if this is what
we always want. The ticket is not completely clear about this in my opinion.
Perhaps Julian can tell us what would be preferred.

[juliandoucette, 2017/01/23 16:59:31]

Good question. A list of required fields is a requirement for help center. A
list of optional fields or mixed required and optional fields is not required
for help center. I don't have a strong opinion about if and how you should
implement optional fields.

[juliandoucette, 2017/01/24 01:05:47]

On that topic, I might suggest accepting a function to resolve True/False here.
But, again, that is not a requirement for Help Center.

[juliandoucette, 2017/01/24 01:10:29]

- Replace "field" with "value" in my response above
- I don't have a strong opinion about if **or** how to implement optional filter
values**

(Sorry for the confusion)

[Vasily Kuznetsov, 2017/01/24 09:24:46]

Ok, I think we're not quite on the same page here so perhaps I formulated my
question badly. Here's an example to illustrate what I mean:

page1:
topic=philosophy

page2:
topic=philosophy,religion

Now if we invoke get_pages_metadata(topic='philosophy,religion'), do we want to
get the metadata from page1 and page2 or only from page2?

Implementation is easy either way, I was just wondering what would be more
useful for you.

[juliandoucette, 2017/01/24 19:32:55]

I think I understood your question correctly.

Help center requires "only from page 2".

[Vasily Kuznetsov, 2017/01/25 11:18:19]

Ok, in this case I misunderstood your previous answer. But now it's clear.

+        if isinstance(metadata[filter_name], list):
+            if isinstance(filter_value, basestring):
+                filter_value = [filter_value]
+            for option in filter_value:
+                if str(option) not in metadata[filter_name]:
+                    return False
+        elif filter_value != metadata[filter_name]:
                 return False
     return True