Initial commit

aa.wiki was previously merged with aa.core. This is the result of the
seperation of the core with the wiki part, aiming to make it easier to
build on top of aa.core and the many modules made for active archives.

.gitignore

+*.pyc
+# db files, sqlite & RDF hashes
+*.db
+*.bak
+*~
+.svn
+*.swp
+*.orig
+venv/
+repositories/
+run/media/

FAQ

+# How can I embed an RSS feed?
+
+Create a link to the feed with an embed relation, like this:
+
+    [[ embed::http://url/of/the/rss/feed.rss ]]
+
+# How can I put an image in the background?
+
+Because Active Archives wiki syntax is based on Markdown, it is possible to use
+plain HTML in your documents, including css directives. If you want to put an
+image in the background of your page you can put somewhere in the source of the
+page the following snippet:
+
+    <style>
+    body {
+        background: url('http://repo.oralsite.stdin.fr/Jerome%20Bel/Jerome%20Bel_show_wrk_01.jpg');
+        -moz-background-size: auto 100%; /* Firefox 3.6 */
+        background-size: auto 100%;
+        background-repeat: no-repeat;
+        background-position:center;
+    }
+    </style>
+
+For more information about the styling properties and their options, you can
+refer to <http://www.w3schools.com/cssref/default.asp>

INSTALL

+FIXME: The instruction in this file are outdated since the refactoring of
+aacore.
+
+
+This is the INSTALL file for Active Archives (AA). This document provides
+convenient steps so a user/developer can have AA running quickly.
+
+
+Dependencies
+============
+
+Please first install these software in order to get AA running properly:
+
+  - Django += 1.3 <http://www.djangoproject.org/>
+  - SQLite3 <http://www.sqlite.org/>
+  - PySQLite2 <http://trac.edgewall.org/wiki/PySqlite>
+  - Git <http://git-scm.com/>
+  - Git-Python <http://gitorious.org/git-python>
+  - Google-diff-match-patch <http://code.google.com/p/google-diff-match-patch/>
+  - Python html5lib <http://code.google.com/p/html5lib/>
+  - Python markdown += 2.1 <http://www.freewisdom.org/projects/python-markdown/>
+  - Python dateutil <http://labix.org/python-dateutil>
+  - Pygraphviz <http://networkx.lanl.gov/pygraphviz/>
+  - Python librdf (Redland) <http://librdf.org/>
+
+On Ubuntu, this should do the trick:
+
+    sudo apt-get install python-pip
+    sudo pip install django
+    sudo pip install gitpython
+    sudo pip install html5lib
+    sudo pip install markdown
+    sudo pip install diff-match-patch
+    sudo pip install python-dateutil
+    sudo pip install pygraphviz
+    sudo apt-get install python-librdf
+
+Additionally, you'll need the following for the RDF sniffers:
+
+  - Exiftool
+  - Ffmpeg
+  - Imagemagick
+
+
+Quick Installation
+==================
+
+1. Clone the repository onto your machine
+
+       git clone git@git.constantvzw.org:aa.core.git 
+
+1. Once you've installed the required dependencies:
+
+       cd /path/to/aa.core/run
+
+2. Build the database:
+
+       python manage syncdb
+
+   The prompt will ask for the admin infos and fixtures will be loaded.
+   
+3. Run the django webserver:
+
+       python manage.py runserver
+
+4. Configure the project domain name at:
+
+       <http://localhost:8000/admin/sites/site/1/>
+
+   Typically the value must be "localhost:8000" if you are running the project
+   on a local server.
+
+
+vim: ft=markdown:

README

+The Active Archives Video Wiki inverts the paradigm of uploading resources into
+a centralized server and instead allows resources to remain "active", in-place
+and online. Caching and proxy functionality allow (light-weight) copies of
+resources to be manipulated and preserved even as the original sources change
+or become (temporarily) unavailable.
+
+Strategically, the project aims to clarify some of the "cloudy" aspects of Web
+2.0 regarding issues of licensing, sharing, ownership, access, and longevity of
+online material. Designed to break open the "black box" of online video, users
+are encouraged to write with video, creating new compositions made from
+collages of disparate (online) elements.
+
+This is alpha software with known bugs. It runs, and works at least some of the
+time, but use at your own risk.
+
+For more information about the project, see <http://activearchives.org/>

TODO

+High priority
+=============
+
+- Skip tag
+- Create new page: default content that give clues
+    - First page directly created?
+    - Give a location to the first annotation
+    - add annotation button: predefined location + meaningful text too.
+* Improve history 
+    * Be able to revert a page
+- layer palette doesn't refresh
+
+
+Low priority
+============
+
+- BUG: negative positions
+- bug with column ":" in url: sould be escaped by the wikify function.
+- shift to enable the grid snapping
+- javascript scrollbars (less ugly)
+- draggable help
+- extra menu to show/hide annotation by criterias
+- Metadata in the sidebar
+- edit locks?
+- button to remove the about attribute/ drag the @ to remove the about attribute + confirmation
+- "attach" boxes
+    - with data-anchor, data-position (north, south, east, west etc.)
+- configure serve to serve best html5 video/audio
+- RSS feeds
+- Still a problem with non consecutive header levels...
+- list pages filter ?
+
+Ideas
+=====
+
+- html5 audio/video: preload="auto"?
+- How can RDF be useful for collective commenting and annotating?
+- rdf metadata to create relation between pages. EG. translation.
+- move page
+- How to preserve the title attribute with the new syntax ([[embed::ressource]])?
+- manual sectioning "8<---"? What about the <br /> tag which is semantically correct for this?
+- Notion of focus for media, to know which one is being controlled
+- different types of landmarks... what syntax?
+
+Oralsite workshop remarks
+=========================
+
+- video leads text/text leads video
+- embed webpages = iframe?
+
+- A nice sniffer: `file`:
+
+    ➜  ~  file /tmp/marqueurs.txt    
+    /tmp/marqueurs.txt: ASCII English text, with very long lines
+
+- annotation negative position happens when one rescales the annotation "outside" the screen.
+- manual to update
+- how to change the embed wrapper?
+
+Meeting 01/2012
+===============
+
+- drop-down css don't work on time-based medias
+
+Navigation
+----------
+
+- how to come back to TOC when sidebar is closed
+- TOC links weird
+- Bug when resizing a box it goes up
+- home page different than publication homepage
+- cinematic mode shows hidden boxes in edit mode
+- drag and drop classes
+- more explicit class names
+- show sound files in the timeline
+- background appears in cinematic mode?
+- remove decorations with classes
+- play/read mode: choose to put a border or not on the boxes
+- if timeline, mode name = play, otherwise mode name = read.
+- possibility to keep titles
+- play/pause/skip to next landmark
+- use the timeline to fly to a box (already working)
+
+Bug
+---
+
+- carriage return on titles
+- history: date sections not accurate
+- Jonathan Burrows on Scores: play mode: sidebar opened the images overflow
+- lightbox: missing navigation buttons
+- renumber section when one gets deleted
+- Question marks in wiki links are mistakenly decoded
+    [[What's the Score Now?]] --> http://localhost:8000/pages/What%27s_the_Score_Now/?/edit/
+- Pages names must not contain non-ascii characters
+
+
+Meeting 27/01/2012
+==================
+
+- Namespace for Ritz
+
+
+LGRU
+====
+
+- sidebar opened by default.
+- editable stylesheets
+- file upload
+- Rester en contact avec Bachir pour l'hébergement + donner accès à Active Archives. Serveur des laboratoires.

aawiki/admin.py

+from django.contrib import admin
+from models import Page
+
+
+class PageAdmin(admin.ModelAdmin):
+    list_display = ("name", "content")
+    search_fields = ("name", "content")
+admin.site.register(Page, PageAdmin)

aawiki/audacity.py

+# This file is part of Active Archives.
+# Copyright 2006-2011 the Active Archives contributors (see AUTHORS)
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+# Also add information on how to contact you by electronic and paper mail.
+
+
+import re
+from aawiki.mdx.mdx_sectionedit import (TIMECODE_HEADER, spliterator)
+from aawiki.timecode import (timecode_fromsecs, timecode_tosecs)
+
+
+def audacity_to_srt(data, explicit=False):
+    """
+    >>> data = '''90,610022  90,610022   first section
+    ... 345,271874  345,271874 second section'''
+    >>> print(audacity_to_srt(data).strip())
+    00:01:30.610 --> 
+    <BLANKLINE>
+    first section
+    <BLANKLINE>
+    00:05:45.272 --> 
+    <BLANKLINE>
+    second section
+
+    >>> data = '''90,610022  345,271874   first section
+    ... 345,271874  512,573912 second section'''
+    >>> print(audacity_to_srt(data).strip())
+    00:01:30.610 --> 
+    <BLANKLINE>
+    first section
+    <BLANKLINE>
+    00:05:45.272 --> 00:08:32.574
+    <BLANKLINE>
+    second section
+
+    >>> data = '''90,610022  345,271874   first section
+    ... 345,271874  512,573912 second section'''
+    >>> print(audacity_to_srt(data, explicit=True).strip())
+    00:01:30.610 --> 00:05:45.272
+    <BLANKLINE>
+    first section
+    <BLANKLINE>
+    00:05:45.272 --> 00:08:32.574
+    <BLANKLINE>
+    second section
+
+    >>> data = '''90,610022  345,271874   first section
+    ... 345,271874  512,573912'''
+    >>> print(audacity_to_srt(data).strip())
+    00:01:30.610 --> 00:05:45.272
+    <BLANKLINE>
+    first section
+    <BLANKLINE>
+    00:05:45.272 --> 00:08:32.574
+    <BLANKLINE>
+    second section
+    """
+    stack = []
+
+    for line in data.splitlines():
+        try:
+            (start, end, body) = tuple(line.split(None, 2))
+        except ValueError:
+            try:
+                # A marker without label
+                (start, end) = tuple(line.split(None, 1))
+                body = ""
+            except ValueError:
+                # A blank line? Get lost!
+                break
+
+        start = float(start.replace(',', '.'))
+        end = float(end.replace(',', '.'))
+
+        start = timecode_fromsecs(start, alwaysfract=True, 
+                                  alwayshours=True, fractdelim=',')
+        end = timecode_fromsecs(end, alwaysfract=True, 
+                                alwayshours=True, fractdelim=',')
+
+        # If the end time equals the start time we ommit it.
+        if end == start:
+            end = ""
+
+        if not explicit:
+            # Deletes previous end time if equal to actual start time
+            if len(stack) and stack[-1]['end'] == start:
+                stack[-1]['end'] = ""
+
+        body = body.replace(r'\n', '\n')
+
+        stack.append({'start': start, 'end': end, 'body': body})
+
+    template = "{e[start]} --> {e[end]}\n\n{e[body]}\n\n"
+    return "".join([template.format(e=e) for e in stack])
+
+
+def srt_to_audacity(data, force_endtime=False):
+    """docstring for srt_to_audacity"""
+    # FIXME: UnicodeDecodeError...
+    pattern = re.compile(TIMECODE_HEADER, re.I | re.M | re.X)
+
+    stack = []
+    for t in spliterator(pattern, data, returnLeading=0):
+        m = pattern.match(t[0]).groupdict()
+
+        if force_endtime:
+            if len(stack) and stack[-1]['end'] == '':
+                stack[-1]['end'] = timecode_tosecs(m['start'])
+            end = timecode_tosecs(m['end']) or ''
+        else:
+            end = timecode_tosecs(m['end']) or timecode_tosecs(m['start'])
+
+        stack.append({
+            'start': timecode_tosecs(m['start']),
+            'end': end,
+            'body': t[1].strip('\n'),
+        })
+
+    template = "{e[start]}\t{e[end]}\t{e[body]}\n"
+    return "".join([template.format(e=e) for e in stack])
+
+
+if __name__ == "__main__":
+    import doctest
+    doctest.testmod()
+    f = open('/tmp/bla.srt', 'r')
+    data = f.read()
+    f.close()
+    #print(audacity_to_srt(data))
+    print(srt_to_audacity(data))

aawiki/filters.py

+# This file is part of Active Archives.
+# Copyright 2006-2012 the Active Archives contributors (see AUTHORS)
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+# Also add information on how to contact you by electronic and paper mail.
+
+
+import urlparse
+import urllib2
+import html5lib
+import lxml
+import re
+from aacore import rdfutils
+import subprocess
+import os.path
+import feedparser
+from utils import get_rdf_model
+from textwrap import dedent
+
+
+class AAFilter(object):
+    def __init__(self, arguments, stdin):
+        self.arguments = arguments or ""
+        self.parsed_arguments = {}
+        self.stdin = stdin
+        self.stdout = stdin.copy()
+        self.stdout['local_path'] = self.get_next_path()
+        self.stdout['local_url'] = self.get_next_url()
+        if self.validate():
+            return self.run()
+
+    def validate(self):
+        return True
+
+    @staticmethod
+    def uri_to_path(uri):
+        return urlparse.urlparse(uri).path
+
+    def get_next_path(self):
+        extension = os.path.splitext(self.stdin['local_path'])[1]
+        return "%s|%s:%s%s" % (self.stdin['local_path'], self.__class__.__name__, 
+                               self.arguments, extension)
+
+    def get_next_url(self):
+        extension = os.path.splitext(self.stdin['local_url'])[1]
+        return "%s|%s:%s%s" % (self.stdin['local_url'], self.__class__.__name__, 
+                               self.arguments, extension)
+
+
+class AAFilterEmbed(AAFilter):
+    name = "embed"
+
+    def run(self):
+        embed_style = None
+        if self.arguments:
+            embed_style = self.arguments
+        else:
+            q = dedent("""\
+                PREFIX dc:<http://purl.org/dc/elements/1.1/>
+                PREFIX aa:<http://activearchives.org/terms/>
+                PREFIX http:<http://www.w3.org/Protocols/rfc2616/>
+                PREFIX media:<http://search.yahoo.com/mrss/>
+                
+                SELECT ?ctype ?format ?audiocodec ?videocodec
+                WHERE {{
+                  OPTIONAL {{ <%(URL)s> http:content_type ?ctype . }}
+                  OPTIONAL {{ <%(URL)s> dc:format ?format . }}
+                  OPTIONAL {{ <%(URL)s> media:audio_codec ?audiocodec . }}
+                  OPTIONAL {{ <%(URL)s> media:video_codec ?videocodec . }}
+                }}""".strip() % {'URL': self.stdin['original_url']})
+
+            model = get_rdf_model()
+
+            b = {}
+            for row in rdfutils.query(q, model):
+                for name in row:
+                    b[name] = rdfutils.rdfnode(row.get(name))
+                break
+
+            # TODO: move to templates
+            if b.get('ctype') in ("image/jpeg", "image/png", "image/gif"):
+                embed_style = "img"
+            elif b.get('ctype') in ("video/ogg", "video/webm") or (b.get('videocodec') in ("theora", "vp8")):
+                embed_style = "html5video"
+            elif b.get('ctype') in ("audio/ogg", ) or (b.get('audiocodec') == "vorbis" and (not b.get('videocodec'))):
+                embed_style = "html5audio"
+            elif b.get('ctype') in ("text/html", ):
+                embed_style = "iframe"
+            elif b.get('ctype') in ("application/rss+xml", "text/xml", "application/atom+xml"):
+                embed_style = "feed"
+            else:
+                embed_style = None 
+
+        # TODO: move to templates
+        if embed_style == "img":
+            self.stdout['output'] = '<img src="%s" />' % self.stdin['local_url']
+        elif embed_style == "html5video":
+            #self.stdout['output'] = '<video class="player" controls src="%s" />' %  self.stdin['local_url']
+            # Temporarily fixes the local serveur serving ogg with the wronf mimetype
+            self.stdout['output'] = '<video class="player" controls src="%s" />' %  self.stdin['original_url']
+        elif embed_style == "html5audio":
+            #self.stdout['output'] = '<audio class="player" controls src="%s" />' % self.stdin['local_url']
+            # Temporarily fixes the local serveur serving ogg with the wronf mimetype
+            self.stdout['output'] = '<audio class="player" controls src="%s" />' % self.stdin['original_url']
+        elif embed_style == "iframe":
+            self.stdout['output'] = '<iframe src="%s"></iframe>' % self.stdin['local_url']
+        elif embed_style == "feed":
+            feed = feedparser.parse(self.stdin['local_url'])
+            self.stdout['output'] = u''
+            for entry in feed['entries'][:4]:
+                self.stdout['output'] += u'<div>'
+                self.stdout['output'] += u'<h3><a href="%s">%s</a></h3>' % (entry.link, entry.title)
+                self.stdout['output'] += u'<div>'
+                self.stdout['output'] += entry.summary
+                self.stdout['output'] += u'</div>'
+                self.stdout['output'] += u'</div>'
+        else:
+            self.stdout['output'] = "<p>Unable to detect embed type</p>"
+
+
+class AAFilterXPath(AAFilter):
+    """ Takes a url as input value and an xpath as argument.
+    Returns a collection of html elements
+    usage:
+        {{ "http://fr.wikipedia.org/wiki/Antonio_Ferrara"|xpath:"//h2" }}
+    """
+    name = "xpath"
+
+    #def validate(self):
+        #sizepat = re.compile(r"(?P<width>\d+)px", re.I)
+        #match = sizepat.match(self.arguments)
+        #if match:
+            #self.parsed_arguments['width'] = match.groupdict()['width']
+            #return True
+
+    def absolutize_refs (self, baseurl, lxmlnode):
+        for elt in lxml.cssselect.CSSSelector("*[src]")(lxmlnode):
+            elt.set('src', urlparse.urljoin(baseurl, elt.get("src")))
+        return lxmlnode
+
+    def run(self):
+        request = urllib2.Request(self.stdin['original_url'])
+        request.add_header("User-Agent", "Mozilla/5.0 (X11; U; Linux x86_64; fr; rv:1.9.1.5) Gecko/20091109 Ubuntu/9.10 (karmic) Firefox/3.5.5")
+        stdin = urllib2.urlopen(request)
+        htmlparser = html5lib.HTMLParser(tree=html5lib.treebuilders.getTreeBuilder("lxml"), 
+                                         namespaceHTMLElements=False)
+        page = htmlparser.parse(stdin)
+        p = page.xpath(self.arguments)
+        if p:
+            self.stdout['output'] = "\n".join([lxml.etree.tostring(self.absolutize_refs(self.stdin['original_url'], item)) for item in p])
+            #self.stdout['output'] = "\n".join([lxml.etree.tostring(self.absolutize_refs(self.stdin['original_url'], item), encoding='utf-8') for item in p])
+        else:
+            self.stdout['output'] = "<p>Your query for %s did not return any elements</p>" % self.arguments
+
+
+class AAFilterBW(AAFilter):
+    name = "bw"
+    def run(self):
+        if not os.path.exists(self.stdout['local_path']):
+            cmd = 'convert -colorspace gray %s %s' % (self.stdin['local_path'], 
+                                                      self.stdout['local_path'])
+            p1 = subprocess.Popen(cmd.split(" "), stdout=subprocess.PIPE, 
+                                  stdin=subprocess.PIPE)
+            (stdout_data, stderr_data) = p1.communicate()
+            #(stdout_data, stderr_data) = p1.communicate(input=self.stdin)
+        self.stdout['output'] = "Conversion successful. Use the embed filter to display your ressource in the page"
+
+
+class AAFilterResize(AAFilter):
+    name = "resize"
+
+    def validate(self):
+        sizepat = re.compile(r"(?P<width>\d+)px", re.I)
+        match = sizepat.match(self.arguments)
+        if match:
+            self.parsed_arguments['width'] = match.groupdict()['width']
+            return True
+
+    def run(self):
+        if not os.path.exists(self.stdout['local_path']):
+            cmd = 'convert -resize %s %s %s' % (self.parsed_arguments['width'], 
+                                                self.stdin['local_path'], 
+                                                self.stdout['local_path'])
+            p1 = subprocess.Popen(cmd.split(" "), stdout=subprocess.PIPE, 
+                                  stdin=subprocess.PIPE)
+            #(stdout_data, stderr_data) = p1.communicate(input=self.stdin)
+            (stdout_data, stderr_data) = p1.communicate()
+
+        self.stdout['output'] = "Conversion successful. Use the embed filter to display your ressource in the page"
+
+
+class AAFilterCrop(AAFilter):
+    """
+    [[ http://example.org/image.jpg||crop:40x30+10+10 ]]
+    """
+    name = "crop"
+
+    def validate(self):
+        sizepat = re.compile(r"(?P<width>\d+)x(?P<height>\d+)\+(?P<top>\d+)\+(?P<left>\d+)", re.I)
+        match = sizepat.match(self.arguments)
+        if match:
+            self.parsed_arguments['width'] = match.groupdict()['width']
+            self.parsed_arguments['height'] = match.groupdict()['height']
+            self.parsed_arguments['top'] = match.groupdict()['top']
+            self.parsed_arguments['left'] = match.groupdict()['left']
+            return True
+
+    def run(self):
+        if not os.path.exists(self.stdout['local_path']):
+            cmd = 'convert -crop %sx%s+%s+%s %s %s' % (self.parsed_arguments['width'],
+                                                       self.parsed_arguments['height'], 
+                                                       self.parsed_arguments['top'], 
+                                                       self.parsed_arguments['left'], 
+                                                       self.stdin['local_path'], 
+                                                       self.stdout['local_path'])
+            p1 = subprocess.Popen(cmd.split(" "), stdout=subprocess.PIPE, 
+                                  stdin=subprocess.PIPE)
+            #(stdout_data, stderr_data) = p1.communicate(input=self.stdin)