planet

June 22, 2015

Dan Frumin

Darcs binaries for OSX with Homebrew

June 22, 2015 12:33 PM UTC

Recently I’ve updated my Darcs homebrew build to Darcs 2.10. You can install it with

brew install http://darcs.covariant.me/darcs.rb

The formula contains a context (--exact-version) and it is a static binary.


Tagged: darcs, haskell, homebrew

May 13, 2015

Dan Frumin

Hoogle inside the sandbox

May 13, 2015 09:54 PM UTC

Introduction

This is my first post from the (hopefuly fruitful!) series of blog posts as part of my Haskell SoC project. I will spend a great chunk of my summer hacking away on DarcsDen; in addition, I will document my hardships and successes here. You can follow my progress on my DarcsHub.

This particular post will be about my working environment.

The problem

Hoogle is an amazing tool that usually needs no introduction. Understandably, the online version at haskell.org indexes only so many packages. This means that if I want to use hoogle to search for functions and values in packages like darcs and darcsden, I will have to set up a local copy.

Cabal sandboxing is a relatively recent feature of the Cabal package manager, but I don’t think it is reasonable in this day to install from the source (let alone develop) a Haskell package without using sandboxing.

The problem seems to be that the mentioned tools do not play well together out of the box, and some amount of magic is required. In this note I sketch the solution, on which I’ve eventually arrived after a couple of tries.

Using hoogle inside a Cabal sandbox

The presumed setup: a user is working on a package X using the cabal sandboxes. The source code is located in the directory X and the path to the cabal sandbox is X/.cabal-sandbox.

Step 1: Install hoogle inside the sandbox. This is simply a matter of running cabal install hoogle inside X. If you want to have a standard database alongside the database for your packages in development, now is the time to do .cabal-sandbox/bin/hoogle data.

Step 2: Generate haddocks for the packages Y,Z you want to use with hoogle. In my case, I wanted to generate haddocks for darcs and darcsden. This is just a matter of running cabal haddock --hoogle in the correct directory.

Step 3: Convert haddocks to .hoo files. Run the following commands in X/:

.cabal-sandbox/bin/hoogle convert /path/to/packageY/dist/doc/html/*/*.txt

You should see something like

Converting /path/to/packageY/dist/doc/html/Y/Y.txt
Converting Y... done

after which the file Y.hoo appears in /path/to/packageY/dist/doc/html/Y/

Step 4: Moving and combining databases. The hoogle database should be stored in .cabal-sandbox/share/*/hoogle-*/databases. Create such directory, if it’s not present already. Then copy the ‘default’ database to that folder:

cp .cabal-sandbox/hoogle/databases/default.hoo .cabal-sandbox/share/*/hoogle-*/databases

Finally, you can combine your Y.hoo with the default database.

.cabal-sandbox/bin/hoogle combine /path/to/packageY/dist/doc/html/*/*.hoo .cabal-sandbox/share/*/hoogle-*/databases/default.hoo
mv default.hoo .cabal-sandbox/share/*/hoogle-*/databases/default.hoo

And you are done! You can test your installation

$ .cabal-sandbox/bin/hoogle rOwner
DarcsDen.State.Repo rOwner :: Simple Lens (Repository bp) String

For additional usability, consider adding .cabal-sandbox/bin to your $PATH.


Tagged: cabal, darcs, haskell, hoogle

April 20, 2015

Simon Michael

ssh, Darcs Hub vulnerability

April 20, 2015 11:10 PM UTC

I recently learned of a serious undocumented vulnerability in the ssh package. This is a minimal ssh server implementation used by darcsden to support darcs push/pull. If you use the ssh package, or you have darcsden’s darcsden-ssh server running, you should upgrade to/rebuild with the imminent ssh-0.3 release right away. Or if you know of someone like that, please let them know.

darcsden is of course the basis for Darcs Hub. Here’s the announcement I sent to users there a few days ago, with more details.


Hello darcs hub users,

This is Simon Michael, operator of hub.darcs.net, with the first all-darcs-hub-users announcement. You’re receiving this because you have an email address configured in your darcs hub user settings.

Thank you for using darcs hub, and for any feedback/bug reports/patches you may have sent. Usage is growing steadily, and I plan to blog more about it soon at joyful.com.

This email is to announce a recently patched security vulnerability in darcs hub’s SSH server.

Timeline:

3/21: a software developer reports that the haskell “ssh” library used by darcs hub does not check for a valid signature on the public key during authentication. This means it was possible to authenticate as any other ssh user if you knew their public key.

3/21-: I discuss the issue with a small number of core darcs developers and the ssh author.

3/25: A preliminary fix is deployed. We believe this closed the vulnerability.

4/6: A more comprehensive and tested fix is deployed.

4/15: This announcement is sent to current darcs hub users with valid email addresses (714 of 765 users).

4/20: Public disclosure via blog, haskell mail lists and the issue tracker (darcsden #130).

Impact and current status:

We believe the vulnerability is now fixed. But we are not cryptographers - I’m sure the new ssh maintainer would welcome any help from some of those.

We have no reason to believe anyone discovered or exploited the vulnerability. Also, it seems unlikely there’s anything hosted on darcs hub that would attract this kind of attention. darcs hub logs are not good enough to be certain, however. It’s possible I’ll find a way to be more certain by looking at file timestamps or something.

The weakness was present in darcs hub’s ssh server since it went live (and in darcsden.com before that). As mentioned, it was possible to authenticate via ssh as another user if you provided their public ssh key. With ssh access, it’s possible to create, delete, modify or replace any repository in that darcs hub account (but not possible to change user settings in the web app, or to access the system hosting darcshub).

The worst-case scenario we’ve imagined is that a motivated attacker could have authenticated as you and replaced your repo with one that looks just like it, but with patches altered or added, any time since you created the repo on darcs hub (or on darcsden.com, if you moved it from there).

So if you’re paranoid/careful you may want to check the integrity of your repos, eg by reviewing the repo history (“changes” button on the website, “darcs log [-s] [-v]” at the console). If you have more questions about this, you can contact me (simon@joyful.com) and if necessary Ganesh Sittampalam (ganesh@earth.li) privately.

Future plans:

Also:

Some logistical things to be aware of:

Needless to say, I regret the vulnerability and am pleased to have it closed. Of course we are not alone, eg github had their own incident. Thank you very much to all who have been helping with this, especially the original reporter for letting us all know, and Ganesh for providing swift and high quality fixes.

April 19, 2015

Darcs News

darcs 2.10.0 release

April 19, 2015 10:01 PM UTC

Hi all,

The darcs team is pleased to announce the release of darcs 2.10.0.

Downloading

The easiest way to install darcs 2.10.0 from source is by first installing the Haskell Platform (http://www.haskell.org/platform). If you have installed the Haskell Platform or cabal-install, you can install this release by doing:

$ cabal update
$ cabal install darcs-2.10.0

Alternatively, you can download the tarball from http://darcs.net/releases/darcs-2.10.0.tar.gz and build it by hand as explained in the README file.

The 2.10 branch is also available as a darcs repository from http://darcs.net/releases/branch-2.10

Feedback

If you have an issue with darcs 2.10.0, you can report it via the web on http://bugs.darcs.net/ . You can also report bugs by email to bugs at darcs.net, or come to #darcs on irc.freenode.net.

What's new since darcs 2.8.5

New features

User Interface

Performance

Issues resolved in Darcs 2.10

Known issues

These are known new issues in darcs 2.10.0:

March 04, 2015

Darcs News

Darcs News #110

March 04, 2015 12:18 PM UTC

News and discussions

  1. The Darcs 2.10 release is near! Please test the release branch and tell us if you find a bug.
  2. Joachim Breitner has shut down his darcswatch service after 7 years of activity:

Issues resolved (11)

issue822 Ernesto Rodriguez
issue2260 Ganesh Sittampalam
issue2385 Guillaume Hoffmann
issue2410 Guillaume Hoffmann
issue2411 Guillaume Hoffmann
issue2414 Guillaume Hoffmann
issue2418 Guillaume Hoffmann
issue2422 Ganesh Sittampalam
issue2427 Ben Franksen
issue2431 Ben Franksen
issue2432 Ganesh Sittampalam
issue2437 Guillaume Hoffmann

Patches applied (111)

See darcs wiki entry for details.

November 18, 2014

Darcs News

Darcs News #107

November 18, 2014 03:45 AM UTC

News and discussions

  1. Darcs has received two grants from the Google Summer of Code program, as part of the umbrella organization Haskell.org. Alejandro Gadea will work on history reordering:
  2. Marcio Diaz will work on the cache system:
  3. Repository cloning to remote ssh hosts has been present for years as darcs put. This feature has now a more efficient implementation:

Issues resolved (11)

issue851 Dan Frumin
issue1066 Guillaume Hoffmann
issue1268 Guillaume Hoffmann
issue1416 Ale Gadea
issue1987 Marcio Diaz
issue2263 Ale Gadea
issue2345 Dan Frumin
issue2357 Dan Frumin
issue2365 Guillaume Hoffmann
issue2367 Guillaume Hoffmann
issue2379 Guillaume Hoffmann

Patches applied (41)

See darcs wiki entry for details.

Darcs News #109

November 18, 2014 03:45 AM UTC

News and discussions

  1. We are in the feature freeze period of darcs 2.10:
  1. Our two Summer of Code projects ended up two months ago. Marcio and Ale's code will be part of the upcoming new stable version of darcs. In case you missed them, here are the latest posts of Marcio for his project:
  1. Ale's posts:

Issues resolved (7)

issue1514 Guillaume Hoffmann
issue1624 Marcio Diaz
issue2153 Andreas Brandt
issue2249 Mateusz Lenik
issue2380 Owen Stephens
issue2403 Ganesh Sittampalam
issue2409 Ganesh Sittampalam

Patches applied (118)

See darcs wiki entry for details.

August 19, 2014

Ale Gadea

Last Few Weeks

August 19, 2014 05:44 AM UTC

In the last three weeks I was working in a couple of things, unfortunately I couldn't complete any to 100% :

- Finishing the command minimize-context.
- Implement the "show dependencies" for darcsden.
- Solve the issue2405.

Finish the command minimize-context

I can start telling that the command itself is implemented, but making tests I find out a case when making minimize-context fails to update correctly the files of repository. I comment the problem in here, however here goes a little summary.

Some "preconditions" to take into account:
1. If exist a tag, I search for dependencies in the patchs after the tag.
2. If happends that not exist dependencies after the tag, the only patch in the context of the bundle to send is the tag. This helps the command darcs apply to merge the patches and seems good to have at least a "tag patch" dependency.
3. (In relation with 1) If not exist a tag, the search is made in all the repository. One problem here is that with repositories that have huge amount of patches (darcs for example with 11000~ patches) the command maybe not finish of calculate the dependencies. Hopefully, having so many patches without tags seems a little odd. Nevertheless, I suppose that even the search in 700~ patches of 10~ patches to send should have a decent performance.

Well, passing to comment the problem. In some older post I mention the way we calculate the dependencies, so with that in mind, suppose we have to repositories $r_1$, $r_2$ with the following patches, where $m_x$ and $a_x$ represents the $x$-th modification and adding respectively.


$r_1$,$r_2$ = $m_2$ $tag$ $m_1$ $a_1$

suppose now that $r_2$ makes a third modification $m_3$ that adds lines without touch any existing line in the only file adding for $a_1$ and sequentially modified for $m_x$. With the idea of send a bundle to $r_1$. Now if we compute the dependencies to find out what is the context to send. For (1) we need to try to commute $m_2$ with $m_3$, which success but of course the end result is a $m'_3$ and a $m'_2$ because the lines which modifies every one now are different. But beyond that for our definition of dependency we can throw $m_2$, so the final context of the bundle with minimize context is the patch $tag$.

The problem now is that if we make darcs apply of the minimized bundle the merge is made not considering $m_2$ and the final state of the file isn't right. For example, if we are talking about that $m_3$ adds a function in some empty place, this could end in $m_3$ adding the function in the middle of an existing function.

My, more or less, mature thought is something like have Direct Dependency and  Indirect Dependency (the names could be differents $\smile$). The Direct Dependency is a dependency as already we know it and the Indirect Dependency is that if we have,

$p_1$ and $p_2$ patches, and $p'_1$ and $p'_2$  are the results of commute the patches, then if $p_1 \ne p'_1$ we conclude that $p_1$ indirectly depends of $p_2$.

I have almost implemented this idea, but I change task and still missing a couple of details. In particular, I'm not so sure of a couple of things,

- How to correctly compare the patches ($p_1 \ne p'_1$)
- In my unclean implementation I have to carry on $MyEq$ for all over the code and it's very ugly I think.

Finishing with this topic, here is a script (improved by Guillaume) that shows the problem:
Link: http://hub.darcs.net/alegadea/ExamplesRepos
SH: getDepsDoesntAlwaysWork.sh

Implement the "show dependencies" for darcsden

An idea that quickly comes up to my mind is that, considering the last part of the previous topic, since we have two types of dependencies it would be nice generate the graph of dependencies with two types of arrows to differentiate the type of dependency between patches. Talking about the implementation itself, I advanced in the drawing of the graph and some different presentations, but very little about the integration with darcsden, which I think is more or less direct having implemented the $js$ that draw the graph. Would have been nice have more time for finish this but I swap to the task of the following topic.

I hope be able to work in the last two topics in the next few months outside of the GSoC.

Solve the issue2405

I would like to start saying, "what a shame...". I can't solve the issue, I improve a little bit the memory but nothing too much significant. Maybe the good news is that, more o less, I found the "place" in the code where the memory triggers "to the sky" :)

Now, the reality is that I tried so many things that I'm a little bit confused about everything. So, I gonna comment the last feelings and tests that I make and, any question(or idea) for the future person (maybe me) that would try to solve the issue, it would well received.

So things that I tried to change:

- Strict version of Map, Monad, etc.
- More strict version of mapM_, gets and modify.
- Some partials changes in the data type PatchIndex.
- Using darcs as library; differents uses of the functions...

The "place" that I suspect the code has the memory glitch is in $applyPatchMods \rightarrow go (pid, PTouch fn)$I think that my lack of experience dealing with this kind of problem comes to light :) the
bright side is that I learn many things on the way of understand the problem and try different solutions.

Closing, maybe this could be my last post in a long time. I would like to thanks Guillaume and the darcs community in general. It was great to contribute to the development of darcs!

Again, I hope be able to work a little outside the GSoC, in particular in the "dependency problems and tasks".

Thanks!

August 01, 2014

Ale Gadea

Other Week (21-26 July)

August 01, 2014 07:13 AM UTC

Good news!

I almost finish to implement the option minimize-context for the command darcs send, I say almost because making some examples I find out that somethings could go wrong. But before entering in that, I will comment a little the main the test that without minimize-context doesn't work and with the option passes(when the implementation is ready the most likely is I will upload the example and some more in ExamplesRepos), and some "tentatives options" for darcs send that I think could be useful.

So, the minimal example:

Suppose we have repositories $r_1$, $r_2$ and $r_3$; $r_2$ a direct clone of $r_1$ where
originally $r_1$ has only one patch, 

$r_1$,$r_2$ $=$ $p_0$

now, we add a patch with a new file in $r_1$ and make a clone, $r_3$ which leave the repositories like this,

$r_1$,$r_3$ $=$ $p_0$ $p_1$

Suppose that we make a modification in the file that add $p_0$ and make a patch $p'_0$ in $r_1$ and we send a bundle (with darcs send) to $r_2$ and $r_3$. What ends up happening is what one imagine,
the bundle can be applied to $r_3$ but not to $r_2$, this because the bundle has de following "shape":

Some message

patch [Some hash]
Author: ...
Date: ...
  * [The modification to $p_0$]

New patches:

$p'_0$

Context:

$p_1$
$p_0$

Patch bundle hash: [Some hash]

and $r_2$ doesn't have the patch $p_1$ despite the fact that $p_1$ nothing has to do with $p'_0$. Using the option minimize-context the bundle is the same but the "Context" is:

Context:

$p_0$

and now there is no problem.

Now, I find out that reducing the context is not always the "best option" for send a bundle. Here I will expose something that could go wrong. Take for example the darcs-screened repository, in total has 11000~ patches, suppose we add a file and make a bundle (using minimize-context) with only the patch that make the add, the context then is empty. So, if we try to apply this bundle to, say the remote repository, this could never end...
What go wrong?; because the context is empty, making darcs apply [TheBundle] in some point try to merge the entire repository versus the "set" of patches to apply, this is very costly if we have 11000~ patches.

I write "best option" before because with my current solution is not always is better reduce the context, but the final idea is that always be the best option. So, I think is necessary to have some care for example in the last implementation if exist a tag, this tag is added to the context. This simple solution solves the problem and seems correct, on the other hand having more than 600~ or 1000~ patches without a tag seem a little extreme :)

Ending, I'm making some tests of two interesting options that came to mind:

. --last-deps=NUMBER
. --deps-from-tag=REGEXP

The first searches dependencies in the first N patches, and the second search since a given tag.

In conclusion, I'm still doubtful about how solve that problem. More if I have consider the two options. For the end of the week I hope have all this solved.

July 24, 2014

Ale Gadea

Some Week (14-19 July)

July 24, 2014 04:43 AM UTC

Hi all!

I was finishing understanding and implementing the command darcs send --minimize-context using "optimize reorder" when I begin to suspect that doesn't solve the problem described in here. The thing is, despite the fact that the context in the bundle to send is reduced if before we send we make "optimize reorder", this doesn't solve the problem of dependencies. Guillaume finished of evacuate my doubts, and so after read:

[darcs-users] darcs cannot apply some patch bundles
irclog
issue1514 (which is the issue which "replace" issue2044 darcs send should do optimize --reorder)

I convince myself of what needs to be done, and it's calculate the "exact" dependencies of the patches to send so such dependencies be the context in the bundle to send. "Exact" because for big repositories can be very costly and calculate till certain tag seem appropriate.

Now, one concern is the cost of doing the search of dependencies. About this I can first comment some of the things I was doing during the week and later show, what I think are, encouraging examples. So first, maybe the most relevant thing of the week it's the implementation of the command darcs show dependencies with the following "description":

Usage: darcs show dependencies [OPTION]... 
Generate the graph of dependencies.


Options:
              --from-match=PATTERN  select changes starting with a patch matching PATTERN
              --from-patch=REGEXP   select changes starting with a patch matching REGEXP
              --from-tag=REGEXP     select changes starting with a tag matching REGEXP
              --last=NUMBER         select the last NUMBER patches
              --matches=PATTERN     select patches matching PATTERN
  -p REGEXP   --patches=REGEXP      select patches matching REGEXP
  -t REGEXP   --tags=REGEXP         select tags matching REGEXP
              --disable             disable this command
  -h          --help                shows brief description of command and its arguments


till the moment the command returns a graph described in dot language, this can eventually change. But with the current returned value one can do:

$\$$ darcs show dep | dot -Tpdf -o deps.pdf

to draw the graph in a pdf. Finally, in summary to calculate the dependencies I use more or less the idea which describes Ganesh in here.

Moving to the examples is interesting, thinking in the performance of the implementation of darcs send --minimize-context using this approach, to see the followings results:

1. Show the dependencies after the tag 2.9.9 (75 patches)
$ time darcs show dep --from-tag=2.9.9
real 0m0.397s
user 0m0.373s
sys 0m0.026s
darcsDesps299.pdf

2. Show the dependencies after the tag 2.9.8 (133 patches)
$ time darcs show dep --from-tag=2.9.8
real 0m2.951s
user 0m2.865s
sys 0m0.082s
darcsDesps298.pdf

3. Show the dependencies after the tag 2.9.7 (288 patches)
$ time darcs show dep --from-tag=2.9.7
real 0m26.654s
user 0m26.003s
sys 0m0.511s
darcsDesps297.pdf

4. Show the dependencies after the tag 2.9.6 (358 patches)
$ time darcs show dep --from-tag=2.9.6
real 0m39.019s
user 0m38.302s
sys 0m0.666s
darcsDesps296.pdf

5. Show the dependencies after the tag 2.9.5 (533 patches)
$ time darcs show dep --from-tag=2.9.5
real 1m53.730s
user 1m51.343s
sys 0m1.939s
darcsDesps295.pdf

Rushed conclusion, seems the performance is quite good even more if we think that for compute the graph dependencies we calculate the dependencies of "all the selected patches against all the selected patches" and in the case of the option for send would only require to calculate "patches to send against all the selected patches".

July 15, 2014

Ale Gadea

Month of June

July 15, 2014 01:43 AM UTC

Here goes a little summary of what I been doing between late june (9~21) and early july (1~11).

First and easy, I have been documenting Darcs.misplacedPatches (old name chooseOrder), D.P.W.Ordered and D.P.W.Sealed. Something to comment is that the semantics of misplacedPatches, not always can clean a tag doing darcs optimize reorder. For example; Suppose we have a repository, $r_1$ with the following patches;

$r_1$ $=$ $t_{1,0}$ $p_{1,0}$ $t_{1,1}$

here all tags are clean, but if we make another repository, say $r_2$, and we pull from $r_1$ of the
following way

$\$$ darcs pull -a -p $p_{1,0}$ $r_1$ (we want to pull the patch $p_{1,0}$, we assume that the name of the patch is $p_{1,0}$ for the matching with -p option)
$\$$ darcs pull -a $r_1$

so now we have,

$r_2$ $=$ $p_{1,0}$ $t_{1,0}$ $t_{1,1}$

and we see that $t_{1,0}$ is dirty. Doing darcs optimize reorder not reorder nothing. What is going on is that to know what reorder, misplacePatches takes the first tag, in our case $t_{1,1}$, and
"search" for what patches he don't tag. But $p_{1,0}$ and $t_{1,0}$ are tagged by $t_{1,1}$ so there is nothing to reorder despite $t_{1,0}$ is dirty. Therefore there is no way of clean $t_{1,0}$ because misplacePatches always takes the first tag, so if a tag is tagging one or more dirty tags, this tags never be available to get clean.

"Second", using the implementation of "reorder" one can get almost for free the option --reorder for the commands pull, apply and rebase pull. The behavior for the case of pull (for the others commands is the same basic idea) is that our local patches remain on top after a pull from a remote repository, e.i. suppose we have the followings $l$(ocal) and $r$(emote) repositories,

$l$ $=$ $p_1$ $p_2$ $\ldots$ $p_n$ $lp_{n+1}$ $\ldots$ $lp_m$

$r$ $=$ $p_1$ $p_2$ $\ldots$ $p_n$ $rp_{n+1}$ $\ldots$ $rp_k$

where $lp$ are the local patches that don't belong to $r$, and vice versa for $rp$. Make darcs pull, leaves $l$ as follow,

$l$ $=$ $p_1$ $p_2$ $\ldots$ $p_n$ $lp_{n+1}$ $\ldots$ $lp_m$ $rp_{n+1}$ $\ldots$ $rp_k$

meanwhile make darcs pull --reorder, leaves $l$,

$l$ $=$ $p_1$ $p_2$ $\ldots$ $p_n$ $rp_{n+1}$ $\ldots$ $rp_k$ $lp_{n+1}$ $\ldots$ $lp_m$

making more easy to send the $lp$ patches later.

"Third", beginning a new task, implement option minimize-context for command darcs send. Still no much to comment, I have almost finished implementing the option but with some doubts, I hope that for the end of the week have a more "prettier" implementation as well as a better understanding.

June 25, 2014

Darcs News

darcs news #104

June 25, 2014 04:59 AM UTC

News and discussions

  1. Google Summer of Code 2013 has begun! BSRK and José will post updates on their blogs:

Issues resolved (8)

issue2163 Radoslav Dorcik
issue2227 Ganesh Sittampalam
issue2248 Ganesh Sittampalam
issue2250 BSRK Aditya
issue2311 Sebastian Fischer
issue2312 Sebastian Fischer
issue2320 Jose Luis Neder
issue2321 Jose Luis Neder

Patches applied (20)

See darcs wiki entry for details.

darcs news #105

June 25, 2014 04:58 AM UTC

News and discussions

  1. This year's Google Summer of Code projects brought a lot of improvements to darcs and its ecosystem!
  2. Gian Piero Carrubba asked why adjacent hunks could not commute:
  3. We listed the changes that occurred between version 2.8.4 and the current development branch into a 2.10 release page:

Issues resolved (8)

issue346 Jose Luis Neder
issue1828 Guillaume Hoffmann
issue2181 Guillaume Hoffmann
issue2309 Owen Stephens
issue2313 Jose Luis Neder
issue2334 Guillaume Hoffmann
issue2343 Jose Luis Neder
issue2347 Guillaume Hoffmann

Patches applied (39)

See darcs wiki entry for details.

Darcs News #106

June 25, 2014 04:58 AM UTC

News and discussions

  1. Darcs is participating once again to the Google Summer of Code, through the umbrella organization Haskell.org. Deadline for student application is Friday 21st:
  2. It is now possible to donate stock to darcs through the Software Freedom Conservancy organization. Donations by Paypal, Flattr, checks and wire transfer are still possible:
  3. Dan Licata wrote a presentation about Darcs as a higher inductive type:
  4. Darcs now directly provides import and export commands with Git. This code was adapted from Petr Rockai's darcs-fastconvert, with some changes by Owen Stephen from his Summer of Code project "darcs-bridge":

Issues resolved (6)

issue642 Jose Luis Neder
issue2209 Jose Luis Neder
issue2319 Guillaume Hoffmann
issue2332 Guillaume Hoffmann
issue2335 Guillaume Hoffmann
issue2348 Ryan

Patches applied (34)

See darcs wiki entry for details.

Darcs News #108

June 25, 2014 04:57 AM UTC

News and discussions

  1. We have a few updates from the Google Summer of Code projects. Alejandro Gadea about history reordering:
  2. Marcio Diaz about the cache system:
  3. Incremental fast-export is now provided to ease maintenance of git mirrors:

Issues resolved (8)

issue2244 Ale Gadea
issue2314 Benjamin Franksen
issue2361 Ale Gadea
issue2364 Sergei Trofimovich
issue2364 Sergei Trofimovich
issue2388 Owen Stephens
issue2394 Guillaume Hoffmann
issue2396 Guillaume Hoffmann

Patches applied (39)

See darcs wiki entry for details.

June 12, 2014

Ale Gadea

Third Week (02-06 june)

June 12, 2014 04:58 PM UTC

Well, well... Now with the solution already implemented here are a couple of time tests that show the improvement.

For the repository of the issue2361:

Before patch1169
"let it run for 2 hours and it did not finish"

After patch1169
real    0m5.929s
user    0m5.683s
sys     0m0.260s

For the repository generated by forever.sh, that in summarize has 12600~ patches, a bundle unrevert and doing reorden implies move 1100~ patches forward passing by 11500~ patches.

Before patch1169
(Interrupted!)
real    73m9.894s
user    71m28.256s
sys     1m11.439s

After patch1169
real    2m23.405s
user    2m17.347s
sys     0m6.030s

The repository generated by bigRepo.sh has 600~ patches, with only one tag and a very small bundle unrevert.

Before patch1169
real        0m34.049s
user        0m33.386s
sys         0m0.665s

After patch1169
real        0m1.053s
user        0m0.960s
sys         0m0.152s

One last repository generated by bigUnrevert.sh, has 13 patches and a really big bundle unrevert (~10MB).

Before patch1169
real    0m1.304s
user    0m0.499s
sys     0m0.090s

After patch1169
real    0m0.075s
user    0m0.016s
sys     0m0.011s

The repository with more examples is in here: ExamplesRepos.

June 05, 2014

Ale Gadea

Second Week (26-30 may)

June 05, 2014 06:47 PM UTC

Luckily, this week with Guillaume we found a "solution" for the issue 2361. But before of entering in details, let's review how the command darcs optimize --reorder does for reorder the patches.

So, suppose we have the following repositories than, reading it from left to right we have the first patch till the last patch, besides with $p_{i,j}$ we denote the $i$-th patch who belongs to the $j$-th repository, and when we want to specify that a patch $p_{i,j}$ is a tag we write $t_{i,j}$.

$r_1$ $=$ $p_{1,1}$ $p_{2,1}$ $\ldots$ $p_{n,1}$ $p_{n+1,1}$ $\ldots$ $p_{m,1}$

$r_2$ $=$ $p_{1,1}$ $p_{2,1}$ $\ldots$ $p_{n,1}$ $p_{1,2}$ $\ldots$ $p_{k,2}$ $t_{1,2}$ $p_{k+1,2}$ $\ldots$ $p_{l,2}$

where the red part represent when $r_2$ was cloned from $r_1$, and the rest is how each repository was evolved. Now, suppose we make a merge of $r_1$ and $r_2$ in $r_1$ making a bundle of the patches of $r_2$ and appling it in $r_1$. Thus, after the merge we have that

$r_1$ $=$ $p_{1,1}$ $p_{2,1}$ $\ldots$ $p_{n,1}$ $p_{n+1,1}$ $\ldots$ $p_{m,1}$ $p_{1,2}$ $\ldots$ $p_{k,2}$ $t_{1,2}$ $p_{k+1,2}$ $\ldots$ $p_{l,2}$

and we found the situation where the tag $t_{1,2}$ is dirty because the green part is in the middle. And now we are in conditions of finding out how darcs does the reorder of patches.
So, the first task is to select the first tag seeing $r_1$ in the reverse way, suppose $t_{1,2}$ is the first (ie, $p_{k+1,2}$ $\ldots$ $p_{l,2}$ are not tags), and split the set of patches (the repository) in

$ps_{t_{1,2}}$ $=$ $p_{1,1}$ $p_{2,1}$ $\ldots$ $p_{n,1}$ $p_{1,2}$ $\ldots$ $p_{k,2}$ $t_{1,2}$

and the rest of the patch set,

$rest$ $=$ $p_{n+1,1}$ $\ldots$ $p_{m,1}$ $p_{k+1,2}$ $\ldots$ $p_{l,2}$

this is done by splitOnTag, which I don't totally understand yet, so for the moment... simply do the above :) Then, the part that interest us now is $rest$, we want to delete all the patches of $rest$ that exist in $r_1$ and then add them again, causing that they show up to the right. This job is done by tentativelyReplacePatches, which first calls tentativelyRemovePatches and then calls tentativelyAddPatches.

So, tentativelyRemovePatches of $r_1$ and $rest$ makes,

$r_{1}'$ $=$ $p_{1,1}$ $p_{2,1}$ $\ldots$ $p_{n,1}$ $p_{1,2}$ $\ldots$ $p_{k,2}$ $t_{1,2}$

and, tentativelyAddPatches of $r_{1}'$ and $rest$,

$r_{1}''$ $=$ $p_{1,1}$ $p_{2,1}$ $\ldots$ $p_{n,1}$ $p_{1,2}$ $\ldots$ $p_{k,2}$ $t_{1,2}$ $p_{n+1,1}$ $\ldots$ $p_{m,1}$  $p_{k+1,2}$ $\ldots$ $p_{l,2}$


leaving $t_{1,2}$ clean.

Well, all of this was for understanding the "solution" for the issue, we are almost there but before let's look at the function tentativelyRemovePatches. It attempts to remove patches with one special care: when one does darcs revert, a special file is generated, called unrevert in _darcs/patches, which is used for darcs unrevert in case that one makes a mistake with darcs revert. One important difference with unrevert is that unlike all the other files in _darcs/patches, unrevert in not a patch but a bundle, that contains a patch and a context. This context allows to know if the patch is applicable. So when one removes a patch (running for example oblitarete, unrecord or amend) that patch has to be removed from the bundle-revert (bundle of the file _darcs/patches/unrevert). It's now always possible to adjust the unrevert bundle, in which case, the operation continues only if the user agrees to delete the unrevert bundle.

But now a question emerge. Is it necessary to accommodate the bundle-revert in the case of reorder?; the answer is no, and it's because we don't delete any patch of $r_1$ so we still can apply the bundle-revert in $r_{1}''$.

So, finally! we find out that for reorder we need a special case of removing, which doesn't try to update the unrevert bundle. And this ends up being the "solution" for the issue, since the reorder blocks in that function. But! beyond this solves the issue something weird is happening, that is the reason of the double quotes for solution :)

This is more o less the step forward for now. The tasks ahead are, documenting the code in various parts and make the special case for the function tentativelyRemovePatches. On the way I will probably understand more about some of the functions that I mention before so probably I will add more info and rectify whatever is needed.

June 03, 2014

Ale Gadea

Google Summer of Code 2014 - Darcs

June 03, 2014 06:46 PM UTC

Hi hi all!

I have been accepted in the GSoC 2014 :) , as part of the work I'll be writing about my progress. The original plan is have a summary per week (or at least I hope so jeje).

I have already been reading some of the code of darcs and fixing some issues;

Issue 2263 ~ Patch 1126
Issue 1416 ~ Patch 1135
- Issue 2244 ~ Patch 1147 (needs-screening) (not any more $\ddot\smile$)

The details about the project is in History Reordering Performance and Features. Also some issues about the project are;

Issue 2361
Issue 2044

Cheers!

First Week (19-23 may)

June 03, 2014 06:42 PM UTC

Sadly, a first slow week, I lost the monday with problems with my notebook for which I have to reinstall ghc, cabal, all the libraries, etc.. but! in the end this helped :)

The list of taks of the week include:

1. Compile and run darcs with profiling flags
2. Write scripts to generate dirty-tagged big repositories
3. Check memory usage with hp2any for the command optimize --reorder for the
generated repositories and repo-issue2361
4. Check performance difference with and without patch-index
5. Document reorder implementation on wiki
6. Actually debug/optimize reorder of issue2361 (Stretch goal)

1. Compile and run darcs with prolfiling flags

This seems pretty easy at first, but turned somewhat annoying because one have to install all the libraries with the option profiling. So a mini-step-by-step of the my installation of darcs with profiling
flags is (i'm using ubuntu 14.04, ghc-7.6.3 and cabal-install-1.20.0.2) :

- Install ghc-prof package, in my case with sudo apt-get install ghc-prof
- Install depencencies of darcs with enable-library-profiling, doing:
    - $ cabal install LIB --enable-library-profiling ( for each library :) )
    - or setting in ~/.cabal/config, library-profiling: True
- Finaly install darcs with enable-library-profiling and enable-executable-profiling

2. Write scripts to generate dirty-tagged big repositories

About this no much to say, I did some libraries to make the scripts that generates the repositories more straightforward. And I wrote some examples, but still in search of interesting examples. A long the week probably I will add examples, hopefully interesting.

3, 4 and 5 all together and mixed

Now, when finally start to generate the examples repositories and play with hp2ps to check differents things, I started to think about others things and I ended up studing the implementation of the command optimize --reorder, in particular I start to write a version which print some info during the ordering of patches, but for now is very dirty implementation.

November 03, 2013

Simon Michael

darcsum 1.3

November 03, 2013 07:38 PM UTC

darcsum was hanging again, so I made some updates:

And since I came this far, I’ll tag and announce darcsum 1.3. Hurrah!

This release includes many fixes from Dave Love and one from Simon Marlow. Here are the release notes.

Site and ELPA package updates will follow asap. All help is welcome.

September 26, 2013

Simon Michael

darcsden/darcs hub GSOC complete

September 26, 2013 11:48 AM UTC

Aditya BSRK’s darcsden-improvement GSOC has concluded, and I’ve recently merged almost all of the pending work and deployed it on darcs hub.

You can always see the recently landed changes here, but let me describe the latest features a little more:

File history - when you browse a file, there’s a new “file changes” button which shows just the changes affecting that file.

File annotate - there’s also a new “annotate” button, providing the standard view showing which commit last touched each line of the file. (also known as the blame/praise feature). It needs some CSS polish but I’m glad that the basic side-by-side layout is there.

More reliable highlighting while editing - the file editor was failing to highlight many common programming languages - this should be working better now. (Note highlighting while viewing and highlighting while editing are independent and probably use different colour schemes, this is a known open wishlist item.)

Repository compare - when viewing a repo’s branches, there’s a new “compare” button which lets you compare (and merge from) any two public repos on darcs hub, showing the unique patches on each side.

Cosmetic fixes - various minor layout and rendering issues were fixed. One point of discussion was whether to use the two-sided layout on the repo branches page as well. Since there wasn’t time to make that really usable I vetoed it in favour of the less confusing one-sided layout. I think showing both sides works well on the compare page though.

Patch bundle support - the last big feature of the GSOC was patch bundles. This is an alternative to the fork repo/request merge workflow, intended to be more lightweight and easy for casual contributors. There are two parts. First, darcs hub issue trackers can now store darcs patch bundle files (one per issue I think). This means patches can be uploaded to an issue, much like the current Darcs issue/patch tracker. But you can also browse and merge patches directly from a bundle, just as you can from another repo.

The second part (not yet deployed) is support for a previously unused feature built in to the darcs send command, which can post patches directly to a url instead of emailing them. The idea (championed by Aditya and Ganesh) is to make it very easy for someone to darcs send patches upstream to the project’s issue tracker, without having to fork a repo, or even create an account on darcs hub. As you can imagine, some safeguards are important to avoid becoming a spam vector or long-term maintenance headache, but the required change(s) are small and I hope we’ll have this piece working soon. It should be interesting to have both workflows available and see which works where.

I won’t recap the older new features, except to say that pack support is in need of more testing. If you ever find darcs get to be slow, perhaps you’d like to help test and troubleshoot packs, since they can potentially make this much faster. Also there are a number of low-hanging UI improvements we can make, and more (relatively easy) bugs keep landing in the darcs hub/darcsden issue tracker. It’s a great time to hack on darcs hub/darcsden and every day make it a little more fun and efficient to work with.

I really appreciate Aditya’s work, and that of his mentor, Ganesh Sittampalam. We did a lot of code review which was not always easy across a large time zone gap, but I think the results were good. Congratulations Aditya on completing the GSOC and delivering many useful features, which we can put to good use immediately. Thanks!

August 09, 2013

Simon Michael

darcs hub, hledger, game dev

August 09, 2013 10:01 AM UTC

Hello blog. Since last time I’ve been doing plenty of stuff, but not telling you about it. Let’s do a bullet list and move on..

darcsden/darcs hub

hledger

FunGEn & game dev

A sudden burst of activity here.

July 24, 2013

Simon Michael

darcs hub repo stats, hledger balance sheet

July 24, 2013 02:50 AM UTC

Recent activity:

I fixed another clumsy query on darcs hub, making the all repos page faster. Experimented with user and repo counts on the front page. I like it, but haven’t deployed to production yet. It costs about a quarter of a second in page load time (one 50ms couch query to fetch all repos, plus my probably-suboptimal filtering and sorting).

I’ve finally learned how many of those names on the front page have (public) repos behind them (144 out of 319), and how many private repos there are (125, higher than expected!).

Thinking about what is really most useful to have on the front page. Keep listing everything ? Just top 5 in various categories ? Ideas welcome.

Did a bunch of bookkeeping today, which inspired my first hledger commit in a while. I found the balancesheet command (abbreviation: bs) highly useful for a quick snapshot of assets and liabilities to various depths (add –depth N). The Equity section was just a distraction though, and I think it will be to most hledger users for the time being, so I removed it.

July 23, 2013

Simon Michael

hub hacking

July 23, 2013 12:30 AM UTC

More darcs hub activity, including some actual app development (yay):

Added news links to the front page.

Cleaned up hub’s docs repo and updated the list of blockers on the roadmap.

Updated/closed a number of issues, including the app-restarting #58, thanks to a fast highlighting-kate fix by John McFarlane.

Tested and configured the issue-closing commit posthook in the darcsden trunk repo. Commits pushed/merged there whose message contains the regex (closes #([0-9]+)|resolves #([0-9]+)|fixes #([0-9]+)) will now close the specified issue, with luck.

Consolidated a number of modules to help with code navigation, to be pushed soon.

Improved the redirect destination when deleting or forking repos or creating/commenting/closing issues.

Fixed a silly whitespace issue when viewing a patch, where the author name and date run together. I’m still confused about the specific code that generates this - the code I expect uses tables but firebug shows divs. A mystery for another day..

July 22, 2013

Simon Michael

hub speedups

July 22, 2013 12:30 AM UTC

More darcs hub hacking today.

July 21, 2013

Simon Michael

darcsden 1.1, darcs hub news

July 21, 2013 03:00 PM UTC

I’ve been hacking (mostly on darcsden/hub) but not blogging recently. Must get back to the old 45-15 minute routine.


darcsden 1.1 released

darcsden 1.1 is now available on hackage! This is the updated version of darcsden which runs hub.darcs.net, so these changes are also relevant to that site’s users. (More darcs hub news below.)

darcsden is a web application for browsing and managing darcs repositories, issues, and users, plus a basic SSH server which lets users push changes without a system login. It is released under the BSD license. You can use it:

http://hackage.haskell.org/package/darcsden - cabal package
http://hub.darcs.net/simon/darcsden - source
http://hub.darcs.net/simon/darcsden/issues - bug tracker

Release notes for 1.1

Fixed:

New:

Brand new, from the Enhancing Darcsden GSOC (some WIP):

Detailed change log: http://hub.darcs.net/simon/darcsden/CHANGES.md

How to help

darcsden is a small, clean codebase that is fun to hack on. Discussion takes place on the #darcs IRC channel, and useful changes will quickly be deployed at hub.darcs.net, providing a tight dogfooding/feedback loop. Here’s how to contribute a patch there:

  1. register at hub.darcs.net
  2. add your ssh key in settings so you can push
  3. fork your own branch: http://hub.darcs.net/simon/darcsden , fork
  4. copy to your machine: darcs get http://hub.darcs.net/yourname/darcsden
  5. make changes, darcs record
  6. push to hub: darcs push yourname@hub.darcs.net:darcsden --set-default
  7. your change will appear at http://hub.darcs.net/simon/darcsden/patches
  8. discuss on #darcs, or ping me (sm, simon@joyful.com) to merge it

Credits

Alex Suraci created darcsden. Simon Michael led this release, which includes contributions from Alp Mestanogullari, Jeffrey Chu, Ganesh Sittampalam, and BSRK Aditya (sponsored by Google’s Summer of Code). And last time I forgot to mention two 1.0 contributors: Bertram Felgenhauer and Alex Suraci.

darcsden depends on Darcs, Snap, GHC, and other fine projects from the Haskell ecosystem, as well as Twitter Bootstrap, JQuery, and many more.


darcs hub news 2013/07

http://hub.darcs.net , aka darcs hub, is the darcs repository hosting site I operate. It’s like a mini github, but using darcs. You can:

The site was announced on 2012/9/15 (http://thread.gmane.org/gmane.comp.version-control.darcs.user/26556). Since then:

Please try it out, report problems, and contribute patches to make it better.


July 13, 2013

Simon Michael

darcsden cleanup

July 13, 2013 12:31 AM UTC

Back to the dev diary. Last post was 11 days ago, after a two-week opening streak of daily posts. I got blocked on one, then got busy. Press on.

Yesterday I started looking at BSRK Aditya’s GSOC darcsden enhancements, to review and hopefully deploy on darcs hub. So far he has worked on alternate login methods (github/google), password reminder, and darcs pack support (for faster gets).

This is forcing some darcsden cleanup, my first darcsden work in a while aside from routine ops and support tasks. I’m going to release what’s in trunk as 1.1, and then start assimilating the new work by BSRK, Ganesh Sittampalam and anyone else who feels like chipping in. Started putting together release notes and a hub status update.

The support requests seem to be on the rise - more usage ? I also found a good bug today: viewing a certain 1K troff file causes darcs hub’s memory footprint to blow up to 1.5G :)

It would be great to have more functionality (like highlighting) broken out into separate, expendable worker processes, erlang style.

July 07, 2013

Simon Michael

darcsden db thoughts

July 07, 2013 11:00 PM UTC

Spent about half of yesterday setting up Aditya’s darcsden patches on the dev instance of hub.darcs.net, testing them, and exploring db migration issues.

Following BSRK’s instructions, I got the dev instance authenticating via Google’s OAuth servers. Good progress. The UI flow I saw needs a bit more work - eg logging in with google seemed to want me to register a new account. Or, there may be a problem with my setup at Google (wrong callback urls ?) - will have to review it with BSRK.

Schema, migrations

My dev instance has so far been using the same database as the live production instance. This is partly because I don’t yet know how to run a second CouchDB instance, partly to reduce complexity, partly to be able to compare old and new code with the same realistic data set.

This of course can lead to trouble, if old and new code require different schemas. darcsden uses CouchDB, a “schemaless” database, but of course there is an implicit schema required by the application code, even if couch doesn’t enforce one. I got more clarity on this when I noticed my dev instance experiments causing errors on the production app.

New darcsden code may include changes to the (implicit) db schema. In this case, there’s a change to the user’s password field. I need to notice such schema changes, and if I want to exercise them on the dev instance, I should first also install them on the production instance. Or, use a separate couchdb instance. Or, use separate databases in the couchdb instance. Or possibly, use separate views in the couchdb databases ?

Eg, here BSRK made the code nicely read user documents (db records) with the old or new schema. Before testing it on the shared db I should have deployed that patch to production as well as dev.

Looking ahead, is this approach (including code to deal with all old schemas) the best way to handle this ? Maybe. It makes things work and seems convenient, at least for now. But it also reminds me of years working with Zope’s ZODB (a schemaless python object database) and the layers of on-the-fly schema updating that built up, and the uncounted number of runtime bugs hunted down due to schema variations in individual objects.

Schema-less or schema-ful ?

While recovering from this, I learned some more about managing couchdb, schema migration, and current couchdb alternatives.

Couch has some really good and unusual qualities, and I feel I’m only scratching the surface of it’s power. Even so, I’m starting to feel a schema-ful, relational database is a better fit for darcsden/darcs hub. Replacing couch has been a topic of discussion on #darcs for some time, for other reasons. Here are some reasons to replace it:

Some reasons not to:

July 01, 2013

Simon Michael

June review

July 01, 2013 11:00 PM UTC

The beginning of a new month. Here’s a quick update.

No hledger release today as there isn’t much new to ship, following a month with several bugfix releases and otherwise mostly infrastructural work (build and dev tool fixes, wiki styling, site update hook). 8/1 is the likely next release date. Oh, John and I also had a nice voice chat - nice to escape the IRC window isn’t it - reviewing our glorious *ledger plans, and I happily accepted his first hledger patch - thanks John! :)

My free hacking time in recent weeks went more towards darcs:

April 30, 2013

Darcs News

darcs weekly news #103

April 30, 2013 07:27 PM UTC

News and discussions

  1. Darcs will be participating to this year's Google Summer of Code under the umbrella of Haskell.org! If you are interested please consult the ideas page and contact us:
  2. Sebastian Fischer implemented darcs-history, a program to be used as darcs posthook and that tracks patch movements inside of a repository:
  3. Sebastian also suggested the possibility for darcs to easily split and merge patches that are depended upon:
  4. Piyush P Kurur was also interested in some special kinds of deep amend-record:

Issues resolved in the last week (1)

issue2274 Guillaume Hoffmann

Patches applied in the last week (27)

See darcs wiki entry for details.