Planet Darcs

Darcs News: darcs weekly news #56

2010-03-07T09:23:13+00:00

News and discussions

Darcs 2.4 was released! Reinier wrote a comprehensive announcement at:
- http://darcs.net/announce.html
Petr explained the difference of performance between darcs 2.3.1 and 2.4 on optimised pristine repositories:
- http://lists.osuosl.org/pipermail/darcs-users/2010-February/023155.html
Max Battcher released musdex, a tool enabling darcs to version compressed files formats such as docx or odt:
- http://lists.osuosl.org/pipermail/darcs-users/2010-March/023171.html

Issues resolved in the last week (3)

issue1731 Reinier Lamers

bump the hashed-storage dependency once more
http://bugs.darcs.net/issue1731

issue1749 Reinier Lamers

refuse to remove nonempty directories in makeRemovePatch
http://bugs.darcs.net/issue1749

issue1755 Eric Kow

fix apparent hard line break in homepage.
http://bugs.darcs.net/issue1755

Patches applied in the last week (27)

See darcs wiki entry for details.

Max Battcher: Storing Documents in Version Control with musdex

2010-03-03T05:12:11+00:00

I've posted my first (presumably buggy) distribution to PyPI (aka the CheeseShop): musdex (musdex documentation). You should be able to just pip install musdex and try it.

This is part of my lead up to Script Frenzy. I'm gearing up to (hopefully) win Script Frenzy by working on a KeyLimePie project. I've got a game backstory and outline I've been working that is less controversial than some of my earlier stuff, and I think a lot of fun. I've put off some of my KeyLimePie work to focus on tooling so that Script Frenzy is as smooth as I can make it.

To get that "real script feel", plus nicely typeset PDF files to help "win" Script Frenzy, I decided to work in Celtx, which is a free screenwriting tool built on the xulrunner platform. I built a tool to convert from a few simple (and I think reasonably script-like) conventions in Celtx to the YAML KeyLimePie data format that I already have tools to work with. (More proof that I think YAML as the basic data format was a good choice.) I took one of my existing YAML scripts, converted it Celtx and then back, as a good test that things work.

That's where musdex comes in. This is a tool that I've meant to build several times over, but knowing that I'm going to be working in Celtx for a month, I decided to finally build it. The Celtx format, like a number of other common document formats today, is really a zip file of a set of XML and HTML documents. musdex is designed so that I can edit a Celtx document as Celtx appears to see it, and yet (transparently) version control the individual XML and HTML documents in my version control system of choice, darcs. Darcs works well with XML and HTML documents and not so well with binary blobs. The resulting patches from XML and HTML documents are much more interesting to review than binary patches.

There's still more to do with musdex, and I expect to find bugs still. This is an early release and I would be interested in hearing feedback on it. As far as I know, it is the first released tool to do zip document version control.

I think it is good enough for my interests in Celtx version control. I tested a Docx and ODT and the XML files in both were hugely minified and it would be nice to have more linebreaks in the XML before doing much in the way of tracking them in a darcs repository. I think it may still be preferable to binary patches, but not by much and I'd hate to have to debug conflicts in those XML blobs. (Although it might not actually be terrible with my tokdiff tool at hand, come to think of it. I wonder if I should post tokdiff to PyPI...)

the Patch-Tag blog: Prepaid EC2 — a simple way to cut amazon cloud costs when working with contractors

2010-03-03T01:58:16+00:00

Amazon EC2 is great, but it can get expensive, especially when you have an army of contractors doing all sorts of different things.

If you’re not careful, you can wind up creeping into a sort of tragedy of the commons situation where nobody shuts down little used EC2 instances because “that’s somebody else’s job” or simply because they’re not sure if anyone is using it. And since amazon bill doesn’t make it easy to see which server did what, let alone who was mostly using which server, it’s hard to formulate a good strategy for beating back the costs.

Here’s a simple, though somewhat radical solution. Prepay cloud developers for EC2 costs and have them use their own amazon accounts. The new improved amazon panel makes it easy to snapshot AMIs, do EBS volume management, and share these resources privately on a per-user basis. Share what you need to share, paypal the developer some funds and let them get on with it.

Suddenly your army of contractors is keeping a much more watchful eye on that EC2 balance because they don’t want to go back to their boss / client to request another prepaid-EC2 cash infusion more often than absolutely necessary.

This isn’t a silver bullet. If you are a fortune 500 company using “enterprise” cloud, you probably don’t want to start encouraging your sysadmins to start expensing basic server costs to their corporate credit card (actually, why not?…. nah). The prepay-EC2 strategy works best where you have a small well defined task that requires an EC2 environment, for example because setup is complicated and/or unautomated, and the developer is a cloudworker type outsourced contractor that wants to get coding with a minimum of friction. I think this approach could also work in a smallish startup environment casual enough that there is a degree of trust but using enouch computing resources that cost is beginning to matter. Prepaid EC2 might be just the trick for keeping the individual team members mindful of costs. I haven’t actually tried this though.

Minor detail: what do you do when the developer does such a good job minding costs that he completes the task far under the prepaid budget? I say, let them keep the excess!

Nota bene: the patch-tag production server is still in linode, and there are no concrete plans to switch to the cloud, though I am kicking the idea around.

Darcs News: darcs weekly news #55

2010-02-20T09:55:47+00:00

News and discussions

Darcs 2.4 rc2 was released, try it now:
- http://lists.osuosl.org/pipermail/darcs-users/2010-February/023078.html
Benchmarks are starting to come to the wiki:
- http://lists.osuosl.org/pipermail/darcs-users/2010-February/023073.html
- http://wiki.darcs.net/Benchmarks
Finally, the fundraising drive was a success, a big thank-you to all donators!
- http://lists.osuosl.org/pipermail/darcs-users/2010-February/023067.html
- http://www.darcs.net/thanks.html

Issues resolved in the last week (1)

issue1741 Eric Kow

fix --list-options when option has multiple names.
http://bugs.darcs.net/issue1741

Patches applied in the last week (6)

See darcs wiki entry for details.

the Patch-Tag blog: The Happstack Guestbook App

2010-02-19T02:30:05+00:00

The Happstack guestbook example application is the best way to learn how to learn HSP with happstack.

It is really cool, and HSP, haskell’s answer to php, has gotten really significantly better in the past few months.

The only thing is that it is ridiculously hard to find.

Hopefully that link three paragraphs back will help.

the Patch-Tag blog: The Witching Hour at Starbucks

2010-02-18T04:24:02+00:00

There is something eerie about the starbucks where I sit programming, around 7 pm. It closes at 9 pm, yet it feels closed already. Employees push mops around. The temperature drops. The light seems a little dimmer.

It is the witching hour.

Out of nowhere, the six or seven people left here start singing happy birthday to Berry, who has made us all many an espresso. We’re on pitch. Somebody is harmonizing, and we actually sound pretty good! Birthday girl is genuinely touched.

It’s a little warmer in here.

Darcs News: darcs weekly news #54

2010-02-15T08:11:39+00:00

News and discussions

Petr updated the hashed-storage library, fixing a performance regression in the upcoming darcs 2.4, and gave us new timing scores of different versions of darcs:
- http://lists.osuosl.org/pipermail/darcs-users/2010-February/023019.html
Eric sketched out a roadmap for darcs 2.5, which influenced our plans for the upcoming hacking sprint:
The darcs team is looking for students who want to apply for a Google Summer of Code 2010:
- http://lists.osuosl.org/pipermail/darcs-users/2010-February/023020.html

Issues resolved in the last week (2)

issue1250 Florent Becker

check for newlines in setpref values
http://bugs.darcs.net/issue1250

issue1719 Florent Becker

Patches applied in the last week (14)

See darcs wiki entry for details.

the Patch-Tag blog: EC2 Amis for Gitit / Happstack

2010-02-12T06:20:19+00:00

Patch-Tag uses linode, but I’ve decided to try out EC2 for development purposes, as the hourly pricing model is attractive for server instances that don’t need to persist, and… well, I’m curious.

First off, with a big bow of respect to the amz engineers, I’ll tell ya, EC2 has gotten a lot better lately.

The last time I used aws was 3 or so months ago when I was configuring cloud stuff for a video startup that used a lot of disk, in conjunction with rightscale management console. Everything worked from a technology standpoint, but my gut feeling vis a vis usability was that aws was an awful mess: guis that didn’t work right, command line interfaces with hard to find documentation, inconsistent flags… beh.

https://console.aws.amazon.com

is hugely improved, documentation and help seems to be better organized… everything is just just more polished… better.

Anyways, to get to the meat of my post, I have created two public amis that I hope may prove useful to developers and newbies alike.

EC2 Amazon Machine Image: ami-3122ce58

AMI ID: ami-3122ce58
Name: Haskell on EC2
Description: Based on elastic karmic, with haskell platform
Source: 072945664613/Haskell on EC2

and… a few cabal installs later

EC2 Amazon Machine Image: ami-3d23cf54

AMI ID: ami-3d23cf54
Name: Gitit Happstack Demo on EC2
Description: Gitit/Hapstack on EC2
Source: 072945664613/Gitit Happstack Demo on EC2

Now, these images are really not such a bit deal since what with cabal and the haskell platform, there have been huge gains in package installability. But they weren’t completely trivial either, due to missing dependencies, the quickcheck 1 to quickcheck 2 switch that broke a lot of happstack, and various other hoops I had to jump through…

*I* had to jump through.

*You* just load the AMIs.

I recommend to start here if you’re an aws newbie.

Happy hacking!

PS Check out the latest from Simon Marlow for another haskell on EC2 post

In case there any aws experts reading this, here is also a snip of a chat of me fumbling around. Feel free to comment or add your questions.

<patch-tag> I am creating an EBS bootable AMI, which takes (at least) > 10
minutes. Say five minutes after start I do “touch somefile”. Will
“somefile” be included in the new AMI? What governs this behavior?
<patch-tag> (I actually am performing this experiment at the moment but I am
curious as to what to expect and why)
<flaccid> um i havnt played with ebs boot yet, but i would assume so, its
meant to be persistent
<flaccid> try it out
<patch-tag> The reason I am asking is, say I want to snapshot while
configuring something, and I am at a safe/happy place. If the
snapshot applies to that exact moment in time, I can keep doing
dangerous things. But if the snapshot extends in time I have to
sit on my hands till it completes before continuing work.
<flaccid> well a snapshot is a snapshot
<patch-tag> er… ?
* flaccid looks up ebs boot
<patch-tag> not particular to ebs boot. question applies to any way of
creating amis.
<patch-tag> (sorry for obfuscating the issue)
<flaccid> oh right
<flaccid> well an AMI is indeed a snapshot
<flaccid> thus why it is called an image
<flaccid> looking at ebs boot, it does seem to be only the ability to boot
from ebs snapshots, so its not a persistent image but a manual way
of gaining persistence
<flaccid> and it costs more
<patch-tag> changing subject — costs more? I thought ebs storage when offline
was same as S3, 10c/G/month.
<patch-tag> also, AMI creation completed, so I’m actually going to check if
that file I created five minutes in got bundled…
<patch-tag> aaaaaand… it’s not there.
<patch-tag> I guess what’s getting snapshotted is ram in a certain state, not
hard disk. which is why it can be truly instantaneous.
<flaccid> everyone has said that

http://aws.amazon.com/about-aws/whats-new/2009/12/03/amazon-ec2-instances-now-can-boot-from-amazon-ebs/

works out costing more
<flaccid> patch-tag: where was the file on the filesystem you touched ?
<patch-tag> /home/ubuntu/testfile
<flaccid> afaik by default /home is not bundled
<flaccid> you will need to check that
<flaccid> and obviously ensure /home is actually on the root fs (/)
<patch-tag> no, it’s bundled. I know this from a previous snapshot.
<patch-tag> any insight on why boot from ebs costs more? I guess because if
you make a lot of snapshots those 10c/G/month add up?
<patch-tag> And the convenience of it makes it tempting to do more snapshots?
<flaccid> well if /home is bundled then, i have no idea. bundling is
bundling…
<flaccid> and yeah can’t answer your second question. unfortunately i’m not
amazon sorry
<patch-tag> just in case it’s not clear, we’re in good shape. this is the
behavior I want.
<patch-tag> My assumption is that snapshot is point in time, and I can
initiate dangerous configs after initiating snapshot without
fear.
<flaccid> like i was saying before, yes a snapshot is a snapshot
<flaccid> whether it is a bundled ami or ebs snapshot, they are both snapshots

the Patch-Tag blog: https repos for patch-tag

2010-02-08T23:30:27+00:00

Patch-tag is now https safe, so you can access your projects with cryptographic security. Just use https style urls when interacting with patch-tag.

By the way — for you code shops with mission critical, high-value, national security endangering repos hosted at patch-tag — https isn’t completely safe.

You could, for instance, get your password sniffed by a man in the middle attack with arp poisoning, if you are in a public network like an internet cafe, or on a trusted network with a coworker who enjoys hanky panky.

There is a simple thing you can do to foil the terrorists though — just click outside the text fields and type some gibberish when you are entering your password.

Ha! Take that script kiddies!

Please enjoy secure repos at patch tag, and as always…

Happy tagging.

************************

UPDATE: Please note the comments which contradict some of my claims. I think I may be wrong about some of this stuff, and am taking a backtrack to make sure my understanding is solid. Watch this space.

Darcs News: darcs weekly news #53

2010-02-01T18:07:43+00:00

News and discussions

Beta 3 was released: ``When no important bugs are discovered in the coming week, darcs 2.4 will be released around Sunday, February 7.''
- http://lists.osuosl.org/pipermail/darcs-users/2010-January/022969.html
- http://lists.osuosl.org/pipermail/darcs-users/2010-January/022992.html
The new hunk editing feature (with an improved user interface in beta 3) spawned other discussions:
- http://lists.osuosl.org/pipermail/darcs-users/2010-January/022981.html
- http://lists.osuosl.org/pipermail/darcs-users/2010-January/022983.html

Issues resolved in the last week (0)

Patches applied in the last week (0)

See darcs wiki entry for details.

the Patch-Tag blog: Taking your gitit wiki offline

2010-01-23T15:55:53+00:00

One of the great things about distributed version control is that you can unplug from the internet and keep checking in.

If you recreate your gitit wiki offline, you can unplug from the internet and keep interacting with your repository via your offline gitit wiki.

Here’s a newsgroup discussion that describes how to use your gitit wiki offline

Disclaimer: If you are not technically savvy, and using patch-tag only for the gitit wiki, this option is probably not for you. I have some ideas percolating though that might make this a lot more straightforward if having an offline gitit wiki is something people want.

Darcs News: darcs weekly news #52

2010-01-23T14:40:28+00:00

News and discussions

darcs 2.4 beta 2 was released, including the fix for performance regression with check and repair:
- http://lists.osuosl.org/pipermail/darcs-users/2010-January/022917.html
Fundraising update: $782 were given, still $1218 to go!
- http://lists.osuosl.org/pipermail/darcs-users/2010-January/022915.html
- http://darcs.net/donations.html

Issues resolved in the last week (1)

issue1731 Reinier Lamers

bump hashed-storage dependency to 0.4.5
http://bugs.darcs.net/issue1731

Patches applied in the last week (15)

See darcs wiki entry for details.

Darcs News: darcs weekly news #51

2010-01-18T09:10:19+00:00

News and discussions

Mark Stosberg shared his remarks concerning the user interface of hunk splitting in darcs 2.4 beta 1, and several improvements were proposed:
- http://lists.osuosl.org/pipermail/darcs-users/2010-January/022838.html
Thanks to our benchmark suite, a performance regression for check and repair was detected, and Reiner Lamers blocked the current release process until this regression is fixed:
- http://lists.osuosl.org/pipermail/darcs-users/2010-January/022868.html
- http://lists.osuosl.org/pipermail/darcs-users/2010-January/022855.html

Issues resolved in the last week (1)

issue1677 Petr Rockai

require hashed-storage 0.4.4.
http://bugs.darcs.net/issue1677

Patches applied in the last week (9)

See darcs wiki entry for details.

the Patch-Tag blog: The Patch Tag Bug Tracker

2010-01-17T18:19:35+00:00

Patch Tag has a bug tracker, an anonymously editable gitit wiki hosted at patch-tag.

Full details at the mailing list, where I made the announcement.

Darcs News: darcs weekly news #50

2010-01-11T16:19:12+00:00

News and discussions

Darcs 2.4 beta 1 was released:
- http://lists.osuosl.org/pipermail/darcs-users/2010-January/022719.html
Eric Kow launched the second fundraising call to prepare the incoming hacking sprint in March ($2000 by 15 feb):
- http://lists.osuosl.org/pipermail/darcs-users/2010-January/022755.html
- http://darcs.net/donations.html

Issues resolved in the last week (2)

issue121 Ganesh Sittampalam

add --ask-deps support to amend-record
http://bugs.darcs.net/issue121

issue1720 Dave Love

Fix cabal haddock problem.
http://bugs.darcs.net/issue1720

Patches applied in the last week (5)

See darcs wiki entry for details.

Max Battcher: Further Adventures with Token Stream Diffs

2010-01-06T01:14:29+00:00

I've taken, with a tiny bit of prodding, the Token Stream Diffs Using Pygments from toy to a nascent toolchain that may even almost be useful.

I've brought in two new dependencies Google diff-match-patch and, for nice argument parsing, argparse. The diff-match-patch library provides a character-based diff algorithm and patch format (a character-based unidiff-like format with character escaping) in a number of languages, including my friend Python. I can use diff-match-patch to produce useful patch output (and apply said patches with a simple new tokpatch.py file that is but a wrapper around diff-match-patch patching).

tokdiff.py has grown three new output formats. The original "toy" format I've renamed "verbose" and its quite interesting for debugging and getting an idea of why diffs look the way they do. Most useful, and the new default, is the unidiff-like output. There's also diff-match-patch's much more compact tab-delimited "delta" format, which is interesting, but I don't think is all that safe. (It's an undocumented, outside of the code itself, feature...)

The final output format is the "compare" which outputs some pretty HTML visually showing the differences between the tokenized diff approach and diff-match-patch's standard character-based diff, plus some basic benchmarking of the two algorithms.

Both tools and both dependencies can be grabbed from the darcs repository:

darcs get http://repos.worldmaker.net/tokdiff/main tokdiff

I'll consider putting together a deeper code site for it in the near future.

Some brief observations and thoughts for future directions:

I shouldn't be too surprised by it, but the tokenized diff does generally seem to be an order of magnitude faster than diff-match-patch's more generalized character-based diff algorithm.
I think there are still some interesting heuristics that can be further applied to make the tokenized diff even smarter. I'm not sure how exactly to start on that (I've been lucky to get as far as I have on the backs of existing diff algorithms).
I'd like to experiment with darcs-like patch selection UI using the tokenized diffs; particularly using the existing tokenization for syntax highlighting.
I'd like to know if anyone finds interesting real applications of this quick hack.

Darcs News: darcs weekly news #49

2009-12-27T21:51:12+00:00

News and discussions

Reinier Lamers announced the release schedule of darcs 2.4 (freeze: December 31st, release: January 30th):
- http://lists.osuosl.org/pipermail/darcs-users/2009-December/022597.html
Eric Kow asked the team to contribute benchmarks showing the performance improvements of the next stable version:
- http://lists.osuosl.org/pipermail/darcs-users/2009-December/022599.html
- http://lists.osuosl.org/pipermail/darcs-users/2009-December/022631.html
Ganesh Sittampalam summed up his work in progress regarding versions 2.4 and 2.5:
- http://lists.osuosl.org/pipermail/darcs-users/2009-December/022603.html

Issues resolved in the last week (2)

issue1362 Joachim Breitner

Mention repo name in mail send body
http://bugs.darcs.net/issue1362

issue1392 <script type="text/javascript"> </script><noscript>caitt@users.sourceforge.net</noscript>

use parsec to parse .authorspelling
http://bugs.darcs.net/issue1392

Patches applied in the last week (32)

See darcs wiki entry for details.

Darcs News: darcs weekly news #48

2009-12-20T18:57:28+00:00

News and discussions

The darcs team discussed the inclusion of Dave Love's darcs Emacs plugin into the contrib directory of darcs:
- http://lists.osuosl.org/pipermail/darcs-users/2009-December/022518.html
Ganesh submitted a patch (based on a previous one by Sergei Trofimovich) making darcs compilable with GHC 6.12:
- http://lists.osuosl.org/pipermail/darcs-users/2009-December/022540.html
Cabal 1.8 and cabal-install 0.8 were released, which will enable non-redundant compilation of darcs and libdarcs:
- http://hackage.haskell.org/package/Cabal-1.8.0.2
- http://hackage.haskell.org/package/cabal-install-0.8.0

Issues resolved in the last week (2)

issue835 Luca Molteni

show file with file directory arguments
http://bugs.darcs.net/issue835

issue1712 Luca Molteni

darcs over-reports number of patches to pull
http://bugs.darcs.net/issue1712

Patches applied in the last week (9)

See darcs wiki entry for details.

Darcs News: darcs weekly news #47

2009-12-12T16:16:17+00:00

News and discussions

Jason Dagit told us about his experiments to make darcs more memory-efficient:
- http://lists.osuosl.org/pipermail/darcs-users/2009-December/022517.html
A discussion occured about the (currently inexistent) handling of symlinks in darcs:
- http://lists.osuosl.org/pipermail/darcs-users/2009-December/022432.html
- http://wiki.darcs.net/Darcs3Wishlist

Issues resolved in the last week (8)

issue223 Benedikt Huber

add --remote-darcs flag for remote darcs executable
http://bugs.darcs.net/issue223

issue1216 Trent W. Buck

list Match section in ToC.
http://bugs.darcs.net/issue1216

issue1308 Luca Molteni

push should warn about unpulled patches before patch-selection
http://bugs.darcs.net/issue1308

issue1336 David Markvica

poor error message on --last "" (empty string to numbers parser)
http://bugs.darcs.net/issue1336

issue1377 Benedikt Huber

getProgname for local darcs instances
http://bugs.darcs.net/issue1377

issue1424 David Markvica

1424: darcs get wrongly reports "using lazy repository" if you ctrl-c old-fashioned get
http://bugs.darcs.net/issue1424

issue1594 Dave Love

define PREPROCHTML in makefile
http://bugs.darcs.net/issue1594

issue1709 quiark@centrum.cz<noscript></noscript>

Short version of progress reporting was wrong.
http://bugs.darcs.net/issue1709

Patches applied in the last week (52)

See darcs wiki entry for details.

the Patch-Tag blog: Transliterating python to haskell: Fibonacci in the state monad

2009-12-07T20:40:17+00:00

I have been doing some python brush-up over the last couple of days, and came across the following bit of code in an irc chat about approaches for fibonacci in python.


def fib(n):
    a, b = 0, 1
    for _ in xrange(n):
        a, b = b, a + b
    return a
print fib(1000)

I think this is pretty typical, reasonable, and efficient python code.

I wondered, how would I express this in haskell? How different this computation looks from the canonical

fibs = 1:1:zipwith (+) fibs (tail fibs)

The answer, as is commonly the case for code written in imperative languages, is that this algorithm lives in the state monad. It can be ported to haskell, practically transliterated, as:


import Control.Monad.State
{-
-- haskell translational of python algo
def fib(n):
    a, b = 0, 1
    for _ in xrange(n):
        a, b = b, a + b
    return a
print fib(1000)
-}
fib :: Integer -> Integer
fib n = flip evalState (0,1) $ do
  forM (xrange n) $ \_ -> do
    (a,b) <- get
    put (b,a+b)
  (a,b) <- get
  return a 

xrange n = [0..(n-1)]

-- test that it works
traditionalFibs' = 1: 1: (zipWith (+) traditionalFibs' (tail traditionalFibs'))
traditionalFib n = traditionalFibs' !! (n-1)

t = fib 1000 == traditionalFib 1000

Also added this to hawiki at

http://www.haskell.org/haskellwiki/The_Fibonacci_sequence#With_state

the Patch-Tag blog: Clearing up loose ends

2009-12-02T23:14:16+00:00

Patch-Tag is getting closer to a place where I’d feel comfortable charging for it, but there are a couple of things I need to take care of first.

First, I want to be more confident that I can scale this service out if I start getting a lot more users. It needs to be rock solid. No more downtime, no more crashes. This will definitely involve code tweaks for places that use unjustifiably large amounts lot of memory, and may (or may not — haven’t decided yet) mean infrastructure changes.

Second, Patch Tag already does backups but I am going to revisit this process one more time to convince myself that this is as rock solid as it can get.

Happy belated thanksgiving everyone. There will be some more interesting news soon, I am feeling sure!

Darcs News: darcs hacking sprint 3 report

2009-11-19T23:36:40+00:00

The Darcs Hacking sprint was co-located in Vienna, Austria and Portland, Oregon, USA.

The Vienna team had a unique experience as the sprint happened to coincide with a student protest that resulted in an occupied University building. Great fun was had by all as we shared our hacking space with the Occupiers on Saturday.

The Portland team was graciously hosted by Galois. As a special treat, we got a visit from Thomas Hartman, one of the folks behind Patch-tag.

Overview

Among other things, the upcoming Darcs 2.4 release has some nice performance improvements from Petr Ročkai's hashed-storage work.

One first goal in this sprint was to polish up this work, making our hashed repository support good enough for GHC team to upgrade from the deprecated old-fashioned format. We think we've largely accomplished what we set out to do. Read on for more details!

Our second goal was to provide a space for new Darcs hackers to get started with making contributions to the project. We had 14 new Darcs developers at this sprint! We hope that they will stay on with the project and continue submitting patches.

Issues resolved

Note that some of this work is still pending patch review or amendements.

issue540 - darcs remove --recursive - Roman Plášil
issue835 - show files with arguments - Luca Molteni
issue1122 - get --complete UI - David Markvica
issue1208 - trackdown --bisect - Matthias Fischmann, Radoslav Dorcik
issue1224 - darcs convert on darcs-2 repositories - Tomáš Caithaml
issue1377 - hardcoding of darcs executable name - Benedikt Huber
issue1392 - use Parsec for authorspellings - Tomáš Caithaml
issue1394 - show time elapsed after each test - Kim Wallmark
issue1499 - versioned show files - Thomas Hartman
issue1500 - misleading darcs progress reports - Roman Plášil
issue1624 - break up global cache into subdirectories - Luca Molteni
issue1643 - optimize --upgrade should do optimize - Christian Berrer

Hacking

Studying darcs

Thomas DuBuisson and Jason Dagit studied some hanging merge cases on the GHC repository to see if we could garner some insights into them. While this did not yield much fruit, it did result a few code cleanups.

Jonathan Daugherty and Josh Hoyt took a tour of the Darcs source. We hope they'll be joining us as Darcs hackers in the future :-)

Jason Dagit produced a concrete and low-level description of the Darcs record command.

Code cleanups

As a warm-up exercise, everybody worked on tidying the Darcs and hashed-storage source, eliminating annoying GHC warnings and implementing suggestions from hlint.

Tomáš Caithaml simplified the darcs show authors authorspellings code by rewriting it in Parsec.

Jason Dagit cleaned up the elegant_merge function and removed commutex in favor of commute.

Jonathan Daugherty submitted some improvements to the user manual.

Performance improvements

Luca Molteni improved Darcs's handling of the global cache, splitting it into buckets for better performance.

User interface improvements

Tomáš Caithaml got darcs convert to abort in case user accidently tries to convert a darcs 2 repository a second time.

Roman Plášil improved the user experience behind Darcs progress reporting replacing misleadingly definitive text with a more accurate description as shown in the following example :

                           OLD TEXT    NEW TEXT
Fetching pristine cache... 4/45        4 done, 41 queued
(sometime later)...        10/166      10 done, 155 queued
(finally)...               297/297     297 done

Christian Berrer extended the darcs optimize --upgrade command to also act as plain old darcs optimize in addition to upgrading the repository format.

David Markvica modified darcs get --complete so that it no longer offers to create a lazy repository.

Benedikt Huber relaxed the assumption that darcs executable would be called "darcs", instead fetching this from the command line.

New features

Matthias Fischmann and Radoslav Dorcik worked on a new trackdown --bisect feature for darcs trackdown which makes trackdown more useful for larger sets sets of patches.

Roman Plášil implemented an oft-requested remove -r feature, making it easier to undo accidental adds of large directories.

Luca Molteni implemented darcs show files with an argument to limit the output to a specific file or directory.

Thomas Hartman implemented matchers support for darcs show files, allowing to get the list of files tracked by Darcs at a given point in the history (eg. darcs show files --tag 2.4).

Petr Ročkai worked on a new more extensible, self-documenting format mechanism as a future alterantive to _darcs/format.

UTF-8 Support

Reinier Lamers continued his work on storing darcs patch metadata in UTF-8. He worked on autodetecting UTF-8 content (in the patch metadata) and found some subtle bugs in the utf8-string library along the way.

SVN integration

The Vienna local team, Christian Berrer, Thomas Danecker and David Markvica, worked on SVN integration and studied the libsvn directory in great detail.

Zero Windows Bugs!

Salvatore Insalaco slayed the 18 Windows bugs that resulted from our recent hashed-storage work. This means that we can can definitely release the new hashed-storage work in Darcs 2.4.

Darcs Team infrastructure

Joachim Breitner made some improvements to the darcswatch core and user interface. He also added hooks to Darcswatch for integration with Darcs Team patch tracker. Joachim Breitner, Eric Kow, Petr Ročkai also added some polish to patch tracker, in particular to its email gateway interface and interaction with the darcs-users mailing list. The result of these efforts is a more efficient Darcs Team with simpler and smoother patch review process.

Kim Wallmark worked on making the 'cabal test' friendlier by outputting the amount of time elapsed after long tests.

Discussions

Priorities for the Sprint

Salvatore Insalaco, Eric Kow, Reinier Lamers, Luca Molteni and Petr Ročkai discussed the key goals for the sprint and also produced a list of the top ProbablyEasy.

Type witnesses

Jason Dagit gave a presentation about our type-witnesses, how the type hackery in darcs is implemented, and why it is useful.

Patch-tag

Thomas Hartman and the Portland crew discussed patch-tag: how it felt from a user perspective and what the most useful features would be.

SVN integration

Thomas Danecker, Eric Kow, Petr Ročkai, Ganesh Sittampalam (IRC) discussed the SVN integration project: what the roadmap should look like and how this feature might fit into Darcs, whether as a fully integrated feature, a plugin or a standalone application.

Petr Ročkai and Eric Kow also discussed the nature of patch/changeset based revision control (like Darcs) and snapshot based revision control (everything else). Petr observed that changeset based version control can be seen as just snapshot based control with a fixed diffing algorithm. To support things like SVN integration, Darcs will need to freeze its diffing algorithm.

Darcs roadmap

Salvatore Insalaco, Eric Kow, Reinier Lamers and all Vienna hackers developed the roadmap for Darcs 2.4 (January 2010) and Darcs 2.5 (July 2010). Using a chalkboard we listed the features we wanted, how desirable they were and what order we should do them in.

Windows support

Salvatore Insalaco, Eric Kow and Reinier Lamers discussed the future of Windows support in Darcs.

Official binary: Our first priority will be to get an official binary for Windows downloadable from Darcs.net. The upcoming Darcs release (Darcs 2.4 in January 2010) will the first to provide an official binaries. Currently, we have the resources to provide a binary for Windows only, but in the future the list of platforms may grow.

Refined integration: After the Darcs 2.4 release, we will work on developing a friendly Windows installer for Darcs which includes not just the Darcs binary but (optionally), TortoiseSSH for better ssh support and a tool for sending email via darcs send.

Cygwin support: We will not be able to support Cygwin, but we will work on providing more explicit documentation on what works (mostly everything) and what does not (absolute cygwin paths). We think that the improved Windows installer will provide a sufficiently comfortable Darcs experience for most Cygwin users.

Fun and speculation

Joachim Breitner, Reinier Lamers and Petr Ročkai discussed abstracting over patch types to just maps (eg. directories), sets and lists (eg. files).

Better Darcs Hacking Sprints

Every sprint teaches something new for what we hope to be a long tradition of biannual Darcs Hacking events.

One thing which worked out well this year was that we provided a clean list of the best ProbablyEasy bugs for new hackers to work on. This allowed new developers to make highly desirable contributions from the very beginning. This year the list was built with live discussion, a few Darcs Team members huddled in a hostel room over an open bug tracker. Perhaps we can replicate the success by having similar pre-sprint meetings in the future.

Replicating success is one thing; how can we do even better? Mainly we need to improve the experience that new hackers have. We were lucky this year to have many new Haskellers and Darcs developers, but we could have done a better job in helping them to get started.

First, some technical issues: A lot of our new developers lost the better part of Saturday morning setting up the machines to build Darcs and to send patches to the list. In the next sprint, we will be ready with much more precise and detailed setup instructions (developed hastily during this sprint) going all the way to the best default settings for Darcs, configuration files for msmtp (or similar sendmail replacements). It also has been suggested that we provide a virtual machine image for instant Darcs hacking.

Second, our mentoring strategy: We were fairly successful at providing in-depth individual mentoring; however what we could done better was to provide more mass mentoring at the very beginning, to help developers send their first patches. To make this work, we should try starting the next sprint with a group mentoring session with the specific purpose of getting people set up to send their first patches. We should encourage new developers to work in small teams, for example with an extra USB keyboard to facilitate pair programming on a single machine.

Attendees

Vienna

Joachim Breitner
Benedikt Huber
Eric Kow
Reinier Lamers
David Markvica
Petr Rockai
Radoslav Dorcik
Salvatore Insalaco
Matthias Fischmann
Thomas Danecker
Christian Berrer
Luca Molteni
Roman Plášil
Tomáš Caithaml
Pivo

Portland

Jason Dagit
Josh Hoyt
Thomas Hartman
Thomas DuBuisson
Kim Wallmark
Jonathan Daugherty

Thanks!

Many thanks to the local team at TU Vienna and Galois for a wonderful welcome! Thanks to the many generous donors who helped us to subsidise travel to the sprint. Thanks also to Microsoft Austria for sponsoring the sprint with drinks and snacks. Molte grazie to Salvatore Insalaco for a wonderful lunch on Sunday.

Finally, thanks to Petr and Luca for the photos, and more generally to everybody who participated in the sprint! It was great to have you and we hope to see you again either on the mailing list or at future sprints.

See you in March!

The fourth Darcs Hacking Sprint will be taking place in Zürich on 19-21 March 2010 as part of the Haskell Hackathon. Hope to see you there!

Darcs News: darcs weekly news #46

2009-11-18T02:34:59+00:00

News and discussions

The third darcs hacking sprint happened last weekend in Vienna and Portland! An extensive report will be posted here soon. As a teaser of all the work done, here is a draft implementation of trackdown --bisect written in Vienna by Matthias Fischmann:
- http://lists.osuosl.org/pipermail/darcs-users/2009-November/022251.html
- http://wiki.darcs.net/Sprints/2009-11
Eric proposed to add Florent Becker to the review team:
- http://lists.osuosl.org/pipermail/darcs-users/2009-November/022270.html
Still working on making patch metadata in UTF8 happen, Reinier is dealing with haskell package version issues for the 2.4 release:
- http://lists.osuosl.org/pipermail/darcs-users/2009-November/022283.html

Issues resolved in the last week (6)

issue540 quiark@centrum.cz<noscript></noscript>

darcs remove --recursive
http://bugs.darcs.net/issue540

issue1122 David Markvica

get --complete should not offer to create a lazy repository
http://bugs.darcs.net/issue1122

issue1224 caitt@users.sourceforge.net <script type="text/javascript"> </script><noscript></noscript>

refuse to convert a repo that's already in darcs-2 format.
http://bugs.darcs.net/issue1224

issue1636 Kamil Dworakowski

match hunk
http://bugs.darcs.net/issue1636

issue1643 Christian Berrer

optimize --upgrade should do optimize
http://bugs.darcs.net/issue1643

issue1659 Petr Rockai

Make restrictBoring take recorded state into account.
http://bugs.darcs.net/issue1659

Patches applied in the last week (34)

See darcs wiki entry for details.

Darcs News: darcs weekly news #45

2009-11-04T15:09:58+00:00

News and discussions

The darcs team is now using roundup as a patch tracker! Eric Kow announced this new feature which now handles all patches send with darcs send:
- http://lists.osuosl.org/pipermail/darcs-users/2009-October/021957.html
Discussion about the formalization of darcs patch theory continued, involving the most prominent darcs theorists around!
- http://lists.osuosl.org/pipermail/darcs-users/2009-October/021970.html
- http://www.math.ucla.edu/~jjacobson/patch-theory/
Reinier Lamers submitted a patch enabling storage of textual patch metadata in UTF-8, fixing one of darcs's oldest bugs. Review pending:
- http://lists.osuosl.org/pipermail/darcs-users/2009-November/022050.html
- http://bugs.darcs.net/issue64

Issues resolved in the last week (2)

issue1554 Trent W. Buck

allow opt-out of -threaded (fix ARM builds).
http://bugs.darcs.net/issue1554

issue1588 Ganesh Sittampalam

add --skip-conflicts option
http://bugs.darcs.net/issue1588

Patches applied in the last week (21)

See darcs wiki entry for details.

the Patch-Tag blog: Parting is such sweet sorrow

2009-11-03T16:36:41+00:00

I’m keeping this one brief… Matt Elder has decided that it’s time to move on from patch-tag, in order to focus on other projects and the new addition to his family.

Thanks for all the good work, Matt. Patch-tag wouldn’t have
gotten this far without you.

the Patch-Tag blog: migrating from happstack 0.1 to happstack 0.3

2009-10-30T14:41:14+00:00

I am in the process of migrating patch-tag to the latest version of happstack, and I thought I would post some diffs to aid others who have the same task.

This probably isn’t of much interest unless you are actually faced with a migration — but if you are, could save you some time and starting at compile complaints.


<                       happstack-data == 0.1,
<                       happstack-helpers == 0.11,
<                       happstack-ixset == 0.1,
<                       happstack-server == 0.1,
<                       happstack-state == 0.1,
<                       happstack-util == 0.1,
---
>                       happstack-data == 0.3.3,
>                       happstack-helpers == 0.30,
>                       happstack-ixset == 0.3.2,
>                       happstack-server == 0.3.3,
>                       happstack-state == 0.3.3,
>                       happstack-util == 0.3.2,

notice msum

< controller pubdom privdom ts dynamicTemplateReload realEmail wikitmpl mime useGoogAnalytics =
<    (staticfiles ++)
---
> controller pubdom privdom ts dynamicTemplateReload realEmail wikitmpl mime useGoogAnalytics =
>     msum [ staticfiles

notice ReaderT

< mainController pubdom privdom ts' dynamicTemplateReload realEmail wikitmpl mime ga =
<        [ ServerPartT $ \rq -> do
---
> mainController pubdom privdom ts' dynamicTemplateReload realEmail wikitmpl mime ga =
>        ServerPartT $ ReaderT $ \rq -> do

notice s/unserverPartT/runServerPartT/

<   unServerPartT ( multi $ mainCommon pubdom privdom (RenderGlobals rq ts mbSession ga) mailFx wikitmpl mime) rq
<   ]
---
>   runServerPartT ( msum $ mainCommon pubdom privdom (RenderGlobals rq ts mbSession ga) mailFx wikitmpl mime) rq

 </code></pre>

the Patch-Tag blog: Prelude golf: what can we learn by rewriting the partition function?

2009-10-28T20:58:11+00:00

To get the most out of the following blog post, first try writing your partition function, which does the same thing as Data.List.partition.

*Partition> partition even [1..10]
([2,4,6,8,10],[1,3,5,7,9])

Then write a testing function, tpf, which checks your partition function against a variety of input, including large or infinite input. (Which means we don’t just use quickcheck).

Then read the blog post to see what obstacles I hit when I went through this process.

source here if you’re irked by wordpress aggressively cutting off the right side of the page.


{-# LANGUAGE NoMonomorphismRestriction #-}
module Partition where
import Debug.Trace.Helpers
import qualified Data.List as DL
import Debug.Trace

-- first attempt
partition f [] = ([],[])
partition f (x:xs) =
  let (as,bs) = partition f xs
  in  if (f x) then (x:as,bs) else (as,x:bs)
t1 = tpf partition 

-- testing function.
tpf partif =
      let a = last . snd . partif even $ [1..(10^6)]
          b = head . snd . partif even $ [1,2..]
          c = last . snd . partif even $ [1..(10^8)]
       in (a,b,c)

-- sanity checks / benchmarks. our own code should be at least as fast as Data.List versions.
-- a is immediate, b is immediate, c takes about 10 seconds.
tDL = tpf DL.partition

-- a test and b test are fine but that 10^8 list seems to take forever.
-- let's think about this. What algorithm are we using?
-- Actually, I don't know the answer to that.
-- but looking at that let/in, I don't have a clear sense of crunching through a list and building up a result, which is usually the ideal.
-- So let's try this with a fold.

-- Next question, what kind of fold?
-- Easy answer: just try both and see what happens!
-- Once you've written one, you have the other just by flipping the arguments in the helper function 
-- (Note: for the foldl, we try strict foldl' by default, as there is usually no reason to use lazy foldl:
-- http://haskell.org/haskellwiki/Foldr_Foldl_Foldl'
partitionR f xs = foldr g ([],[]) xs
  where g next accum@(as,bs) = if f next then (next:as,bs) else (as,next:bs)
partitionL' f xs = DL.foldl' g ([],[]) $ reverse xs
  where g accum@(as,bs) next = if f next then (next:as,bs) else (as,next:bs)
-- Harder answer, which requires thought: we probably want a foldr here, because the b test uses an infinite list and a strict fold
-- won't return anything if you feed it an infinite list.

-- OK, let's try foldr
t2 = tpf partitionR -- result: immediate stack overflow on the a test, and if you try them separately, it overflows all 3 tests.
                    -- that's kind of sad.

-- for no particular reason, let's try a strict foldl version (warning! be careful running this!)
t3 = tpf partitionL'
  -- at least the a test succeeds after a few seconds, confirming the intuition that strict foldl' crunches through a list
  -- and returns output after it's done.
  -- the b test exhausts computer memory, eventually the fan starts whirring louder than I can think, mouse stops responding and have to reboot
  -- so like I said, careful running this, and hit control c!

--okay, I give up, let's peek at the standard libs (SL) Data.List
-- this isn't word for word what the ghc standard libs Data.List has, so as to be more similar to the functions we wrote so far,
-- but it's the same algo, and same performance.
-- the important difference is the lazy match
partitionSL p xs = foldr g ([],[]) xs
  where g next accum = if (p next) then (next:fst accum,snd accum) else (fst accum, next:snd accum)
        -- which is the same as below, using irrefutable pattern syntax:
        -- g next ~accum@(as,bs) = if (p next) then (next:as,bs) else (as, next:bs)
-- Sure enough, same performance as Data.List.Partition
-- Note: to get the same performance as lazy list, you need to compile the module before loading in ghci, eg
-- ghc --make PartitionM.hs, then ghci PartitionM
-- if you do ghci PartitionM.hs, this doesn't terminate (at least, not in under a minute I'm an impatient guy.)
tSL = tpf partitionSL

-- So, why does partition need an irrefutable pattern in the accum argument?
-- Let's look at the original right folded partition again, rewritten as follows, for a 3 element list

partitionR2 f xs = foldr (select f) ([],[]) xs
select f next accum@(as,bs) = if f next then (next:as,bs) else (as,next:bs)
-- select' g next accum = if (p next) then (next:fst accum,snd accum) else (fst accum, next:snd accum) 

tpr2 :: ([Int],[Int])
tpr2 = partitionR even [1..3]
tpr2a = 1 `g` ( 2 `g` ( 3 `g` ([],[]) ) )
g = select even

-- when the accum argument to select is a strict pattern match, the algorithm can't proceed past the first element
-- without evaluating everything inside the first set of parenthesis.
-- For a million element list, this is going to be a problem.
-- if the accum arg is lazy (just accum without the @ binding), the algorithm can proceed to the if test,
-- calculate the tuple with the list parts as just a thunk to be evaluabed later, and keep chugging along,
-- by calculating the list parts next.

Darcs News: darcs weekly news #44

2009-10-28T17:36:41+00:00

News and discussions

Judah Jacobson put online a document formalizing patch theory using inverse semigroups:
- http://lists.osuosl.org/pipermail/darcs-users/2009-October/021886.html
- http://www.math.ucla.edu/~jjacobson/patch-theory/
Jason Dagit announced that the third darcs hacking sprint will also involve a group of hackers in Portland, in parallel with the Vienna meeting:
- http://lists.osuosl.org/pipermail/darcs-users/2009-October/021916.html

Issues resolved in the last week (2)

issue1563 Eric Kow

official thank-you page.
http://bugs.darcs.net/issue1563

issue1652 Eric Kow

suggest cabal update before cabal install.
http://bugs.darcs.net/issue1652

Patches applied in the last week (21)

See darcs wiki entry for details.

the Patch-Tag blog: haskell refactoring technique for naughty house elves: make it compile by making it undefined

2009-10-26T15:30:42+00:00

It is said that in haskell, If it Compiles, It Works.

This is true, but what do you do when it won’t compile?

I came across a real world scenario for this just now, when I was in the process of removing the HStringTemplateHelpers dependency from happs tutorial. The reason for this is that HStringTemplateHelpers uses the unix package (indirectly, via FileManip), which means that at the present time happs tut won’t run on windows, which lately has me smacking myself like dobby the masochistic house elf in the harry potter movies.*

The dependency wasn’t in one file, but scattered throughout the code. Some places, the dependency wasn’t that important, I could just comment the function out, but in other cases the function was core to the app and commenting it out caused cascading compile failures. Doing the type/function arithmetic in my head for figuring out what depended on what was giving me a headache and tempting me to veg out on youtube rather than face the problem, always a warning sign for me that I’m doing something wrong.

I needed to make my program compile, just so I could think about it with my head screwed on straight.

So, instead of commenting out the HStringTemplateHelpers-dependent functions, I set them equal to undefined and commented out their type signatures (since the type sig might be using a type that was defined in the dependency.

-- paintProfile :: RenderGlobals -> String -> UserProfile -> String -> String paintProfile rglobs user cp userimagepath = undefined {-.....-}

A few comment-outs later and cabal install compiles! Of course, if I actually try running the resulting binary, I will hit an undefined error right away, but the point is that I can think again, and start rewriting my offending functions in a methodical way until there are no more undefineds, which is easly checked with

grep -irn undefined src

For what it’s worth, I could have also used

paintProfile rglobs user cp userimagepath = error "paintProfile uses HStringTemplateHelpers... bad dobby! bad dobby! Dobby will have to shut his ears in the oven door for this."

But whatever

* I suppose the righter thing to do would be to fix FileManip, but I am choosing the easy way out just to get it done. Eventually I woud like to move away from HStringTemplate with happs and use mostly Text.XHTML and/or HSP, which is type safe, which HSTringTemplate just thrown in for convenience and newb friendliness.

the Patch-Tag blog: cabal install a package with local haddock documentation

2009-10-23T16:46:26+00:00

I wanted to have a look at the latest happstack release, and noticed the documentation still wasn’t available on haddock.

http://hackage.haskell.org/package/happstack

It wasn’t available locally either, when I did cabal install happstack-server. In fact, haddock was not available offline for anything.

To get the haddocks locally, I edited ~/.cabal/config, setting

Documentation: True

(and uncommenting that line by removing the starting –)

After this, cabal install –reinstall built the documentation in

.cabal/share/doc/happstack-server-0.3.3/html/

handy

But shouldn’t this flag be on by default?

UPDATE:

cabal haddock –hyperlink-source

installs documentation with links to source code, which imho should also be on by defualt.

cabal install –haddock-options=–hyperlink sourceblehblehbleh

doesn’t throw an error. without the blehblehbleh it runs, but doesn’t install with source links, which seems like a bug to me.