Commit 5f6d12eb authored by Michael Murtaugh's avatar Michael Murtaugh
Browse files

makeserver 3 complete?

parent 14ed8af0
Pipeline #597 canceled with stages
......@@ -2,3 +2,4 @@ build/
venv/
*.pyc
*~
makeserver.egg-info/
This diff is collapsed.
---
title: LOG
title: "Makeserver: Change log"
---
Design designs log
=======================
Extended discussion / notes of design decisions
Single makefile
----------------
After encountering a subtle bug with the "cascading makefile" system, making decision to try a simple single makefile (either explicitly specified, or defaulting to ./makefile).
Problem: "index_.html" exists in the subdirectory 2016-10-05. There's also a (local) makefile in 2016-10-05 which is being detected by makeserver. When this makefile is --question'ed, make returns 0 (instead of 2) because make --question of a file which exists will always be 0 when the file in questions is NOT a target of the makefile.
2016
=================
QUESTION the usefulness of multiple levels of makefiles ... it's creating confusion! ... A SIMPLE (but maybe too drastic) solution: Assume a single makefile, default is any one in the root of the webserver. Eventually other processes can be run for sub directories.
It seemed very strong how when generalizing the data join, I suddenly (finally) started to care about the form of the incoming data. In other words, the specific labels of the data started to have an effect as they connected (or didn't) with the names of properties in the rdfa "template" element. Thus the document becomes instrumentalized as a sort of filter / query into the incoming data stream. Now, finally, the benefits of data rewriting via a mechanism like json-ld would be most useful (and appreciated) to complete the loop without a need for ad hoc reframing / tweaking of the data. The hit or miss effect of the selection, filtering and ignoring unnecessary data, is reminiscent (in an encouraging way) of json-ld's contexts as filters.
From Indexical back to dynamic index
-------------------------------------
Static index documents are (1) annoying to leave litered around in each visited directory, (2) inherantly tricky to manage the make rules to regenerate them as needed (on file deletion for instance).... ie now fighting with bad indexes. (3) Git integration will only add to this complexity. That said indexical is still interesting as a separate project for generating / merging data + documents.
A surprising discovery has been the re-finding of writing "template" code in the form of a sample table row with rdfa properties. It seems it might be useful to revisit early web publishing flows that must have used this kind of mechanism (vague memories of Internet Explorer specific html element properties like data-source).
Next step? Data value filters (based on a type? ... or simply connected to a specific value) to do date formating and file size niceness!
By Date
=================
(later: I realize that this data join stuff was best separate from the main line of the makefile based server and can better be split off and developed on it raises quite a few questions on it's own)
24 nov 2016
24 nov
-------------
Woke up this morning with the thought: this project must be *promiscuous*.
......@@ -55,14 +45,14 @@ WRITING WITH CODE
23 nov 2016
23 nov
-----------
* Enabled operation without makefile
* iframe widget
* move fragment code from player to "iframe"(frame) widget
22 nov 2016
22 nov
---------------
* <s>Rehookup index.cgi as directory listing</s> -- actually this became a proper class (no CGI!)
......@@ -78,19 +68,8 @@ WRITING WITH CODE
* 14 Nov 2016: Change to simplified single makefile
* 18 Nov 2016: Added ?edit syntax and "direct editor" view.
earlier
----------
It seemed very strong how when generalizing the data join, I suddenly (finally) started to care about the form of the incoming data. In other words, the specific labels of the data started to have an effect as they connected (or didn't) with the names of properties in the rdfa "template" element. Thus the document becomes instrumentalized as a sort of filter / query into the incoming data stream. Now, finally, the benefits of data rewriting via a mechanism like json-ld would be most useful (and appreciated) to complete the loop without a need for ad hoc reframing / tweaking of the data. The hit or miss effect of the selection, filtering and ignoring unnecessary data, is reminiscent (in an encouraging way) of json-ld's contexts as filters.
A surprising discovery has been the re-finding of writing "template" code in the form of a sample table row with rdfa properties. It seems it might be useful to revisit early web publishing flows that must have used this kind of mechanism (vague memories of Internet Explorer specific html element properties like data-source).
Next step?
* Data value filters (based on a type? ... or simply connected to a specific value) to do date formating and file size niceness!
(later: I realize that this data join stuff was best separate from the main line of the makefile based server and can better be split off and developed on it raises quite a few questions on it's own)
1 Dec 2016
1 Dec
----------------
In explaining the project yesterday, I naturally described the server as the "make server" ... and given that make is now by far the focus of it's operation, this seems indeed the best / most descriptive and honest naming. So I made the change. One interesting consequent is that in typing makeserver to start it, one is reminded, nicedly of the make command. Also: Folded in the submodules including editor (leaving ace-builds a submodule). Need to decide what to do with draggable / player.
......@@ -100,7 +79,7 @@ TODO: Clean the editor code to not require the (rather) unholy mess of both jque
2017
----------
=======
Request.process
res = site.getResourceFor()
......@@ -117,14 +96,14 @@ TODO
* "Sticky" preferences (like theme/fonts/etc)
March 2017
March
-------------
* add ?touch query option + functionality in main code (remove cgi)
* 12 April 2017: Observation: auto "edit" linking functionality would be nice -- but from the index page (ie automagically adding the ?edit, which can then be manually removed) (ie original decision to do this auto editing INSIDE the server was too rigid)
August 2017
August
----------------
Tackling the rather bad problem of path errors with the directory listing.
......@@ -204,7 +183,7 @@ A quite bad consequence of this means that every request gets quite slow as it m
Next Step: Should further integrate the new "Makefile" class into the server (currently it separately implements the actual calls to make) -- in this way force a synchronisation of behaviour between the directory listing and the actual make process.
2018
--------
==========
Thinking to move away from the ?edit way of editing...
1. keep ?edit but map to a shadowable HTML/JS
......@@ -220,36 +199,29 @@ Editor via plugin
* Editors as link modifiers? (javascript means of dynamically processing links?)
2020
----
========
makeserver3
Bumped the version to "3" for an updated implementation based on python3 + aiohttp (replacing twisted).
Makeserver is an inside-out web server.
Makeserver is a scaffolding server that allows dynamic websites to be developed using a makefile.
Requires
* python 3.5
* aiohttp
To do
* (websockets-based) view to monitor output of make command
* add --js option to mixin custom javascript plugins (a la greasemonkey)
* add --saveable option to mixin saveable javascript
makewiki (aka cookbook)
Curated set of deployable makefiles
For instance to convert AVI to webm
Cookbook as a curated set of deployable makefiles: for instance to convert AVI to webm
2021
--------
========
Putting "code folding" into practice, merging many strands into the original project.
Finally I encounter and start to understand/appreciate [Scons](http://scons.org). Thanks in large part to [Vijya Kumar's From Make to SCons](http://www.bravegnu.org/blog/make-to-scons.html).
backend:
python / aiohttp server to serve files, run scons / terminal
......@@ -275,3 +247,21 @@ Some processes take a long time to run (like a video edit, a web crawl, etc). De
scons -Q -n --tree=status
Design decisions
=======================
Why were certain key design decisions made?
Single makefile
----------------
After encountering a subtle bug with the "cascading makefile" system, making decision to try a simple single makefile (either explicitly specified, or defaulting to ./makefile).
Problem: "index_.html" exists in the subdirectory 2016-10-05. There's also a (local) makefile in 2016-10-05 which is being detected by makeserver. When this makefile is --question'ed, make returns 0 (instead of 2) because make --question of a file which exists will always be 0 when the file in questions is NOT a target of the makefile.
QUESTION the usefulness of multiple levels of makefiles ... it's creating confusion! ... A SIMPLE (but maybe too drastic) solution: Assume a single makefile, default is any one in the root of the webserver. Eventually other processes can be run for sub directories.
From Indexical back to dynamic index
-------------------------------------
Static index documents are (1) annoying to leave litered around in each visited directory, (2) inherantly tricky to manage the make rules to regenerate them as needed (on file deletion for instance).... ie now fighting with bad indexes. (3) Git integration will only add to this complexity. That said indexical is still interesting as a separate project for generating / merging data + documents.
# install:
# pip3 install -e .
%.html: %.md
pandoc --toc --css style.css --standalone $< -o $@
......@@ -11,9 +11,14 @@ import base64
Todo: use proper http error responses for errors, see aiohttp/http_exceptions.py
"""
MAKEABLE_UP_TO_DATE = 0
MAKEABLE_NEEDS_REMAKING = 1
MAKEABLE_NOT_MAKEABLE = 2
MAKE = {
'UP_TO_DATE': 0,
'NEEDS_REMAKING': 1,
'NOT_MAKEABLE': 2
}
MAKE_BY_CODE = {code: value for value, code in MAKE.items()}
async def is_makeable (path, cwd=".", makefile="makefile"):
"""
......@@ -26,8 +31,8 @@ async def is_makeable (path, cwd=".", makefile="makefile"):
p = await create_subprocess_exec("make", "--question", "-f", makefile, rpath, cwd = cwd, stdout=DEVNULL, stderr=DEVNULL)
# retcode 0=file is up to date, 1=file needs remaking, 2=file is not makeable
ret = await p.wait()
print ("is_makeable {0}: {1}".format(path, ret), file=sys.stderr)
await log("is_makeable {0}: {1}".format(path, ret))
print (f"is_makeable, makefile:{makefile} path:{path} ret: {MAKE_BY_CODE[ret]}", file=sys.stderr)
await log(f"is_makeable {0}: {MAKE_BY_CODE[ret]}".format(path, ret))
return ret
async def make (path, cwd=".", makefile="makefile", force=False):
......@@ -176,8 +181,8 @@ async def route_get (request):
path = '.'
im = await is_makeable(path, makefile=request.app['makefile'])
remake = 'remake' in request.rel_url.query
if path and (im == MAKEABLE_NEEDS_REMAKING or \
(im == MAKEABLE_UP_TO_DATE and remake)):
if path and (im == MAKE['NEEDS_REMAKING'] or \
(im == MAKE['UP_TO_DATE'] and remake)):
code, resp = await make(path, makefile=request.app['makefile'], force=remake)
if not code:
resppage = """<!DOCTYPE html>
......@@ -279,6 +284,8 @@ def main ():
ap.add_argument("--static", nargs=2, default=None, action="append")
args = ap.parse_args()
print ("Makeserver Version 3.1.0")
if sys.platform == 'win32':
# Folowing: https://docs.python.org/3/library/asyncio-subprocess.html
# On Windows, the default event loop is SelectorEventLoop which does not support subprocesses.
......
......@@ -13,7 +13,7 @@ setup(
license="MIT",
author="Michael Murtaugh",
packages=find_packages(),
install_requires=[],
install_requires=['aiohttp'],
long_description=long_description,
classifiers=[
"Programming Language :: Python",
......
body {
margin: 5%;
}
\ No newline at end of file
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment