The Darcs team is pleased to announce the release of Darcs 2.12.1!

The most important changes are:

• fixes for building with GHC 8

• drop support for GHC 7.6 and 7.8, i.e., require GHC 7.10

• improvements in `darcs whatsnew` output with irrelevant files (Ben Franksen)

This release can be installed via cabal or (soon) stack. The 2.12 branch is also available as a darcs repository from http://darcs.net/releases/branch-2.12 .

The Darcs team is pleased to announce the release of Darcs 2.12.0.

Downloading

One way of installing Darcs 2.12.0 is with stack:

$ stack install darcs-2.12.0

Or first install the Haskell Platform (http://www.haskell.org/platform)
and install Darcs with cabal-install:

$ cabal update
$ cabal install darcs-2.12.0

You can also download the tarball from
http://darcs.net/releases/darcs-2.12.0.tar.gz and build it by hand.

The 2.12 branch is also available as a darcs repository from
http://darcs.net/releases/branch-2.12

What's new

Patch dependency graph export

The new command `darcs show dependencies`, enables to export the dependency graph of a repository (up to the latest tag, by default) as a Graphviz file:

whatsnew output and third-party VC frontend support

The flag `whatsnew --machine-readable` is a simplified version of `whatsnew -s` for easier parsability by third-party tools. Darcs 2.12 adds conflict markers to the output of whatnew when summarized (ie, when used with the `-s` or `--machine-readable` flags or via the `darcs status` alias). Thanks to this, Darcs support was reintroduced in meld 3.15.2 .

improvements for Git repository imports

File moves are converted to file moves primitives, instead of being file deletes and add as before. This enables to have smaller Darcs respositories with a more understandable history. This change adds to other improvements and fixes that make Git import more practical.

repository Weak Hashes

The command `darcs show repo` now shows a hash that is the XOR
of all hashes of the patches metadata of the repository. Being a XOR,
it does not depend on the patches' ordering. Also it is quite fast to
calculate. This Weak Hash can be useful to quickly check whether two
repositories of a single proyect have the same patches.

News and discussions

1. We will release Darcs 2.12 by the end of this month:
   
   • http://lists.osuosl.org/pipermail/darcs-users/2016-April/027198.html
   • http://darcs.net/Releases/2.12
2. On May 6th-8th in Helsinki, a joint sprint Pijul/Darcs is organized:
   
   • http://lists.osuosl.org/pipermail/darcs-users/2016-April/027197.html
   • http://darcs.net/Sprints/2016-05

Issues resolved (5)

issue1807 Guillaume Hoffmann

• clarify help of PAGER, DARCS_PAGER
• http://bugs.darcs.net/issue1807

issue2258 Guillaume Hoffmann

• improve patch index error message with suggestion
• http://bugs.darcs.net/issue2258

issue2393 Guillaume Hoffmann

• remove whatsnew functionality from annotate
• http://bugs.darcs.net/issue2393

issue2486 Ben Franksen

• obliterate --not-in-remote -q should be more quiet
• http://bugs.darcs.net/issue2486

issue2494 Ben Franksen

• output of darcs record with file arguments
• http://bugs.darcs.net/issue2494

Patches applied (96)

2016-04-14 Guillaume Hoffmann

• move network-related tests to network dir, update command names
• resolve issue2393: remove whatsnew functionality from annotate
• add log --machine-readable to see patch dependencies non-interactively
• help of log

2016-04-01 Ganesh Sittampalam

• add some doc comments to RepoType

2016-03-29 Guillaume Hoffmann

• merge Repository.Util into Repository.State
• use B and BC instead of BS and BSC in Repository.State
• fix prelude import in Repository.State
• move maybeApplyToTree to Darcs.Patch.Apply
• move getRecursiveDarcsRepos to UI.Commands.Optimize
• move patchSetfMap to Darcs.Patch.Set
• move functions from Repository.Util to Patch.TokenReplace
• comment in Repository.Util
• refactor similar functions in Darcs.Repository.State
• use readUnrecordedFiltered in getReplaces
• inline a function in Clone
• no longer move index to index.old on mingw32 os
• clarify comments in Darcs.Repository.State
• hlint Darcs.Repository.State
• move External module from Repository to Util
• move Compat and Lock modules from Repository to Util
• merge Darcs.Repository.Ssh into Darcs.Util.Ssh
• remove Darcs.Repository.Read by moving readRepo back to Internal
• add comments and remove checks of optimize commands wrt repo formats
• make all optimize subcommands require hashed except upgrade
• move copySources from HashedRepo to Clone
• move Storage.Hashed modules to Darcs.Util
• remove unused function from Storage.Hashed.Plain
• fix compile error in Storage.Hashed.Test
• remove Storage.Hashed.Utils, move functions to Darcs.Utils.ByteString
• move index-related functions from Utils to Index
• removed unused or redundant functions from Storage.Hashed.Utils
• remove unused functions from Storage.Hashed.Hash
• hlint Storage.Hashed.Darcs
• reuse functions from Darcs.Util.Path
• remove unused Storage.Hashed.Packs

2016-03-09 Ben Franksen

• revert command: be quiet when requested
• accept issue2480: display unicode in patch content
• slightly improved chaotic indentations and import lists
• refactor: use maybeRestrictSubpaths
• refactor: use Darcs.Util.English.capitalize
• replace Darcs.Util.Printer.<> with <> from Data.Monoid; restructured haddocks
• small code layout fix in whatsnew command
• fixed Darcs.Util.English.andClauses and orClauses
• two simple refactorings in the conflict resolution code
• cleanup in revert command: use debugMessage for debug messages
• cleanup: break over-long line in D.R.Merge
• accept issue2494: output of darcs record with file arguments
• resolve issue2494: output of darcs record with file arguments
• refactored some, added readUnrecordedFiltered and maybeRestrictSubpaths
• several fixes and refactorings in fixSubPaths and maybeFixSubPaths
• add Darcs.Util.Printer.quoted and Darcs.Util.Text.pathlist
• added missing hsep function to D.Util.Printer
• added missing Eq and Show instances for ScanKnown
• added Darcs.Util.Printer.ePutDocLn
• add new type IncludeBoring for includeBoring option (was Bool)
• announceFiles only if verbosity /= Quiet

2016-03-05 Guillaume Hoffmann

• rm hashed-storage changelog
• put copyright headers in hashed-storage modules
• add Storage/Hashed dir to checkdeps contrib script
• merge Storage.Hashed.AnchoredPath into Darcs.Util.Path
• explicit exports for Storage.Hashed.Utils
• list and comment exports of Storage.Hashed.Darcs and Plain
• remove Storage.Hashed
• resolve issue2258: improve patch index error message with suggestion
• resolve issue1807: clarify help of PAGER, DARCS_PAGER
• fix extra-source-file path in darcs.cabal

2016-03-07 Ben Franksen

• Darcs.UI.Commands.Unrecord: honor quiet option everywhere
• resolve issue2486: obliterate --not-in-remote -q should be more quiet

2016-02-25 Ganesh Sittampalam

• print the rebase status even after an error
• in runJob, pull repojob out to first-level decision
• refactor displaying suspended status a bit
• inline repoJobOnRebaseRepo
• use helper types to elide more cases in runJob
• elide some common cases in runJob
• reorder runJob cases by job type
• flatten runJob case statements
• add a helper type for flags needed for Rebase
• lift the runJob debugMessage call outside the case
• lift 'therepo' outside the runJob case statement
• express the V1/V2 patch type switch via a GADT too
• use SRepoType to control the rebase type in runJob
• remove commented-out cases for old TreeJob
• drop unnecessary constraints
• break out a runJob function
• drop CarrierType - it can't ever be Rebasing p now
• drop RecontextRebase
• drop NameHack
• inline MaybeInternal module into Named.Wrapped
• make the Rebase import qualified
• Introduce RebaseP to replace Rebasing type
• add 'activecontents' to replace 'patchcontents' for use in conflict resolution
• stop Convert using Wrapped.patchcontents
• add nullary RepoType
• flip dependency between Named.Wrapped and Rebase.Container
• add wrapper type around 'Named'

See darcs wiki entry for details.

News and discussions

1. After 7 years of being the maintainer/Benevolent Dictator of Darcs, Eric Kow stepped down and offered me (Guillaume Hoffmann) to take over, which I accepted:
   
   • http://lists.osuosl.org/pipermail/darcs-users/2015-October/027144.html
   • http://lists.osuosl.org/pipermail/darcs-users/2015-October/027145.html
2. The release process of Darcs 2.12 will start when GHC 8 is released:
   
   • http://lists.osuosl.org/pipermail/darcs-users/2016-March/027185.html
3. We had two new minor releases of Darcs 2.10, and in spite of being minor, they contain a few interesting changes and optimizations:
   
   • http://lists.osuosl.org/pipermail/darcs-users/2015-November/027151.html
   • http://lists.osuosl.org/pipermail/darcs-users/2016-January/027182.html
4. In a span of 4 months we had two sprints, one in Paris in September and another another in Seville in January, check out the reports:
   
   • http://blog.darcs.net/2015/09/darcs-hacking-sprint-9-report.html
   • http://blog.darcs.net/2016/01/darcs-hacking-sprint-10-report.html
5. Finally, Pierre-Étienne Meunier announced a Pijul sprint in may in Finland. Darcs hackers are welcome!
   
   • http://pijul.org/2016/04/02/sprint.htm

Issues resolved (7)

issue2269 Eric Kow

• push hijack test to suspend time
• http://bugs.darcs.net/issue2269

issue2276 Eric Kow

• Keep track of patch hijack decisions
• http://bugs.darcs.net/issue2276

issue2400 Ben Franksen

• use async package to keep track of unpack threads
• http://bugs.darcs.net/issue2400

issue2459 Ben Franksen

• fall back to writing the file if createLink fails
• http://bugs.darcs.net/issue2459

issue2479 Ben Franksen

• root dir most not be among the sources of a move
• http://bugs.darcs.net/issue2479

issue2481 Ganesh Sittampalam

• expose API for 'darcs diff' command
• http://bugs.darcs.net/issue2481

issue2489 Guillaume Hoffmann

• dequote filepaths while importing from git
• http://bugs.darcs.net/issue2489

Patches applied (176)

2016-03-26 Sergei Trofimovich

• allow zip-archive-0.3

2016-03-22 Guillaume Hoffmann

• remove failing-issue1609 from testsuite as property we don't want

2016-03-06 Ben Franksen

• move command: fixed punctuation of error messages and added a comment
• use bug from impossible.h instead of error
• cleanup in Darcs.Patch.Merge: use case instead of fromJust (do return ...)

2016-02-12 Ganesh Sittampalam

• Get rid of the need for DummyPatch in Darcs.Patch.Match

2016-03-05 Guillaume Hoffmann

• move runHLint.sh to root and update to run outside of testsuite

2016-02-12 Ganesh Sittampalam

• drop re-exports from Darcs.Patch.Rebase
• abstract out checking whether a Named is internal
• add versions of simplifyPush[es] for Suspended
• move addFixup to Rebase.Container and give it a clearer name
• add Repair instance for Suspended
• Ignore the rebase internal patch when showing dependencies
• simplify instance ShowPatchBasic (RebaseSelect p)
• break out PatchInspect instance for Suspended
• rename mkSuspended to mkRebase and make it work on 'Suspended'
• use Suspended instead of FL RebaseItem in takeRebase
• break out RepairToFL instance for Suspended
• break out ReadPatch instance for Suspended
• break out Check instance for Suspended
• break out Show instances for Suspended
• break out Conflict instance for Suspended
• break out Effect instance for Suspended
• add PrimPatchBase instance for Suspended
• break out ShowPatch instance for Suspended
• break out Apply instance for Suspended
• refactor instance ShowPatch Rebasing a bit
• inline a couple of defaults to simplify future refactoring
• abstract out an instance ShowPatchBasic Suspended
• Introduce a 'Suspended' type to encapsulate 'FL RebaseItem'

2016-03-18 Ben Franksen

• skip ssh tests if ssh server is down
• made network ssh tests more robust by passing --skip-long-comment
• fix ssh network tests so they work in the test harness
• skip http network tests when server does not respond
• run network tests by default
• resolve issue2479: root dir most not be among the sources of a move
• accept issue2479: bug descending in modifyTree

2016-03-08 Guillaume Hoffmann

• update failing-issue2219
• acknowledge that issue1196 is solved
• acknowledge a working case in failing-index-argument.sh
• merge HACKING into README.md

2016-02-17 Ganesh Sittampalam

• get rid of a couple of trailing newlines

2016-02-05 Guillaume Hoffmann

• remove unused executable and testsuite dependencies
• comment in cabal file workaround
• group all non-optional build-depends
• remove unused darcs-test dependencies
• comment use of flag REENTRANT
• drop definition of HAVE_SIGINFO_H unused since 2009
• hlint Darcs.Util.Diff.Patience
• format patch names within 20 characters in dependencies output
• show dependencies up to last tag by default
• further merge hashed-storage code and tests into darcs code
• bump second html dependency
• darcs show dependencies
• implement function getDeps

2016-01-28 Ganesh Sittampalam

• fix git test involving deletions

2016-01-25 Guillaume Hoffmann

• handle file moves and copies when importing from git
• recommend using -M flag on git fast-export
• tests related to git import of file moves
• use F and T instead of From and To in whatsnew --machine-readable
• bump dependencies lower bounds implied by requiring ghc 7.6
• 2.10.3 changelog

2016-01-29 Ganesh Sittampalam

• support transformers-compat 0.5.x

2016-01-25 Guillaume Hoffmann

• avoid irrefutable pattern when importing unnamed commit
• test for deleting empty directories on git import
• delete empty directories on git import
• rollback filename dequoting on import since now done during parsing
• quoting and escaping of filenames in convert export and import
• test for checking filepaths consistency with git
• resolve issue2489: dequote filepaths while importing from git

2016-01-28 Ganesh Sittampalam

• fix tests that use "git commit"
• applyToTree is just a specialisation of applyToState
• drop unnecessary constraint
• simplify type
• drop unused (and never defined) putApplyState
• move the ObjectMap related code to the FileUUID patch type
• disentangle the state-specific ApplyMonad methods
• swap argument order to ApplyMonad/ApplyMonadTrans
• Rename Prim.V3 to Prim.FileUUID
• move listConflictedFiles out of Conflict class
• Get rid of default implementation of conflictedEffect
• Add some tests for how conflicts are reported
• Drop an unnecessary call to patchcontents in applyAndFix
• Drop unnecessary use of patchcontents in hunkmatch and touchmatch

2016-01-16 Guillaume Hoffmann

• rename Patch and RealPatch to RepoPatchV1 and RepoPatchV2 in harness
• rename Patch and RealPatch to RepoPatchV1 and RepoPatchV2 in darcs code
• do not open patches uselessly when no file restriction given
• add Darcs.Test.Patch.Selection and one unit test
• convert import should be a RepoJob, not a V2Job
• replace TaggedPatch by LabelledPatch in a comment
• whatsnew --machine-readable help string update on file moves
• --machine-readable flag for more parseable whatsnew
• fix code inside of a comment

2016-01-16 Ganesh Sittampalam

• resolve conflicts between changes to splitters and to hijacking
• capture diffAlgorithm in splitters instead of passing it to SelectChanges unconditionally
• drop unneeded export
• simplify the PatchInspect (Rebasing p) instance
• implement hunkMatches for PatchInfoAnd
• move Rebasing out into its own module
• break RebaseItem out into its own file
• bump async dependency
• conditionalise a couple of orphan instances
• resolve conflict in build-tools removal
• drop build-tools restriction
• Bump time, HTTP dependencies

2016-01-15 Guillaume Hoffmann

• set use-time-1point5 flag default to True to please stack
• disable interfering env variable in issue1739 test
• rename README to README.md to get it properly rendered

2016-01-15 Ganesh Sittampalam

• resolve conflict between binary version bump and containers dep
• bump binary, transformers and tar upper bounds

2016-01-14 Guillaume Hoffmann

• make commit an alias for record
• implement repoXor and show it in "show repo" output as "Weak Hash"

2015-12-28 Ganesh Sittampalam

• Portability fix - #type nl_item isn't always Int32
• add test that lost deps during rebase are reported on
• remove unused fmapPIAP

2015-12-22 Guillaume Hoffmann

• fix repo upgrade help string

2015-12-02 Ganesh Sittampalam

• resolve issue2481: expose API for 'darcs diff' command

2015-11-20 Guillaume Hoffmann

• remove a flag needed only for GHC < 7
• remove -fno-warn-dodgy-imports from modules that were still using it
• no longer hide catch from Prelude since we require ghc>=7.6
• acknowledge -fno-warn-dodgy-imports is always needed
• merge Darcs.Patch.ConflictMarking into Darcs.Patch.Conflict

2015-11-29 Ganesh Sittampalam

• bump dependencies on vector, process, HUnit
• force grep to treat output of locale as text

2015-11-20 Guillaume Hoffmann

• Rename Darcs.Repository.LowLevel to Darcs.Repository.Pending

2012-08-09 Eric Kow

• Haddock the pending patch parts of Darcs.Repository.State.
• Make Darcs.Repository.isSimple apply over a whole list.

2015-11-09 Guillaume Hoffmann

• rename NEWS to CHANGELOG to please hackagedb
• fix release date of 2.10.2
• update NEWS for 2.10.1 and 2.10.2
• fix two tests after stopping using the word changes in pull message
• shorter README with quickstart instructions

2015-11-06 Ganesh Sittampalam

• add comments about the rejected 'hasDuplicate' cases
• "Fix" some intermittent QuickCheck failures
• disambiguate imports in some test code
• Add an option to control the number of QuickCheck iterations
• make test-framework imports explicit

2015-11-05 Guillaume Hoffmann

• refactor breakAfterNthNewline and breakBeforeNthNewline
• refactor clone code
• download patches pack asynchronously
• ignore meta- files in packs when cloning
• comment in doOptimizeHTTP

2015-06-28 Ben Franksen

• remove race from D.R.Packs, further simplify the code

2015-10-31 Guillaume Hoffmann

• replace changes by log in release.sh
• remove darcs.spec.in file from 2008
• replace changes by log in Setup.lhs
• update upload.cgi with new command names
• update buildbot-try.sh with new command names
• update cygwin-wrapper file with new commands names and flags
• remove annotate xml schema no longer needed
• remove patch index correctness and timing scripts from contrib
• adapt tests to using patches word instead of changes
• update commands names in help strings

2015-10-28 Ganesh Sittampalam

• split issue1932 test up into a network and non-network part
• Avoid subshells in amend-unrecord test
• disable issue2086 test on Windows - umasks don't really work there
• using mmap on Windows was causing test failures
• warn when suspending "hijacked" patches in rebase pull and apply
• be a bit clearer about patch names in hijack test

2015-09-20 Eric Kow

• resolve issue2269: push hijack test to suspend time
• resolve issue2276: Keep track of patch hijack decisions
• Generalise hijack warning to support use in other commands
• Helper to capitalize a sentence

2015-06-24 Ben Franksen

• resolve issue2459: fall back to writing the file if createLink fails
• resolve issue2400: use async package to keep track of unpack threads
• removed special handling of --to-match from cloneRepository

2015-10-16 Guillaume Hoffmann

• remove redundant import

2015-10-15 Ganesh Sittampalam

• drop sandi lower bound to support GHC 7.4 and add an upper bound

2015-10-03 Daniil Frumin

• Switch from dataenc (deprecated) to sandi

2015-10-09 Guillaume Hoffmann

• replace changes by log in two help strings

2015-09-18 Eric Kow

• Refactor darcs send patch count text snippet
• Tidy darcs send msg code (shorter lines)
• Fix typo in darcs send message

2015-09-19 Guillaume Hoffmann

• make patch selection lazier in presence of matchers
• get rid of selectChanges
• inline patchSetToPatches in the only place where it is used

See darcs wiki entry for details.

Last weekend we had our tenth Darcs sprint, and our first one in Spain. This time indeed, the sprint was organised in the University of Seville, ETSII (Technical Superior School of Informatics Engineering), on January 15th to 17th.



We were 3 participants: Florent Becker, Guillaume Hoffmann and Ganesh Sittampalam. We also had Pierre-Étienne Meunier on video call and Simon Michael on IRC.

Darcs and Pijul integration

One major topic during the whole sprint was the possible integration of Darcs with Pijul. Pijul is a new version control system based on a different patch theory, whose main author is Pierre-Étienne Meunier. Florent also contributes to Pijul and announced its first release last October.

Pijul is promising in terms of how it handles conflicts in a better way than Darcs (better as: better presentation to the user and better performance). There may be a future where Darcs uses Pijul patches by default.  We had many conversations with Florent to understand the internals of Pijul and how it manages patches.

On the first day of the sprint we did a video call with Pierre-Étienne Meunier, to discuss integration of Pijul core with Darcs. It happens that the Darcs code is modular enough to handle Pijul patches (with some work). That afternoon Florent started to work on a Haskell binding for libpijul (through a C binding maintained by Pierre-Étienne, Pijul being implemented in Rust).

Ganesh, Florent and Pierre-Étienne are going to work towards a better integration of both systems. Pierre-Étienne plans to release a 0.2 version of Pijul soon.

Ganesh and Florent with Pierre-Étienne on video call

Renaming Patch/RealPatch to RepoPatchV1/RepoPatchV2

The code of Darcs contains many different layers of patch types. One of them is represented by the two types Patch and RealPatch, and specifies the behaviour of named patch when they are commuted and in case of conflicts. The "Patch" type is the behaviour of patches in repositories with the darcs-1 patch semantics (which can still be created by Darcs) and "RealPatch" is for darcs-2 semantics (the current default of Darcs). I sent a patch to rename these types into something less confusing: RepoPatchV1 and RepoPatchV2.

Interactive selection performance and refactoring

Even if we wrote a patch that improved greatly performance during the last sprint (and we now have a unit test for it), the command "darcs rollback -p ." still remains much slower than "darcs rollback" before presenting the first choice of patch to the user. Florent determined that this was because the action of matching patches within interactive selection is not lazy, ie, the whole list of patches has to be scanned and classified before the first prompt is shown to the user. Florent unearthed a refactor he had of the patch selection code and started rebasing it against the current code.

User manual and developer handbook

We want Darcs to have a user manual again, and a developer handbook that would compile documentation for programmers and computer scientists. We decided the manual should live in darcs' repository itself (so that it stays up-to-date) and the developer handbook on the wiki.

Darcs on Stackage

On IRC, Simon Michael (after an initial request by Joachim Breitner) committed himself to maintain a stack.yaml file for Darcs, and during the weekend Darcs was added to stackage for easier building.

Cleanup, fixes and refactorings

Ganesh tracked down bugs in rebase and sent a few cleanup patches. Moreover he's improving the code of "darcs test" (formerly called "darcs trackdown") so that uncompilable states are neither considered as Passing nor Failing, and bisect is going to be more efficient.

What happens next

I am going to release Darcs 2.10.3 within a couple of weeks, and Darcs 2.12 within a couple of months. This new major version will have optimizations (some of them are already backported to the 2.10 branch) and code refactorings. It may contain the stash feature currently developed by Ganesh. 

This year we hope to have another sprint, and to have more developers participating. Please consult the How to Help, Developer's Getting Started and Projects page on the wiki to get involved!

Ganesh, Florent and Guillaume

After a one year and a half absence, the Darcs Hacking Sprint returned!

Once again, the event occurred at the IRILL (Innovation and Research Initiative for Free Software) in Paris, on September 18th to 20th.

The sprint had 7 participants: Danill Frumin, Eric Kow, Florent Becker, Ganesh Sittampalam, Guillaume Hoffmann, Thomas Miedema and Vinh Dang.

Darcs and GHC 8

Thomas Miedema is a Haskell and GHC hacker, and came on the first day of the sprint. Since Darcs is a system that aims at supporting the various GHC versions out there, Thomas helped us preparing for GHC 8, the next major version. He explained us one issue of GHC 8 that got triggered by Darcs: a bug with the PatternSynonyms extension. Fortunately it seems that the bug will be fixed in GHC HEAD. (First release candidate is planned for December).

Thomas explaining PatternSynonyms to Eric and Ganesh

Diving into SelectChanges and PatchChoices code

• NamedPatch: represent the contents of a patch file in _darcs/patches, that is, its info and its contents
• PatchInfoAnd: represents the info of a patch as read from an inventory file (from _darcs/inventories or _darcs/hashed_inventory) and a lazy field to its corresponding NamedPatch.

Now, getting the NamedPatch for some patch is then obviously more costly than a PatchInfoAnd. You may even have to download the patch file in order to read it (in the case of lazy repositories). Moreover,  the majority of matchers only need the patch info (or metadata), not its actual contents. Only two matchers (hunk and touch) need to actually read the patch file, while matching or a patch name for instance (probably the most common operation) does not.

So, before the sprint, as soon as you wanted to match on a patch file, you had to open (and maybe download) its file, even if this was useless. With my change (mostly in Darcs.Patch.Match) we gained a little more laziness; and the unreasonably slow command "rollback -p ." passes from 2 minutes to ~15 seconds on my laptop. I hope to push this change into Darcs 2.10.2.

Eric, Guillaume and Vinh

Now, the real source of the "rollback -p ." slowness is that patch selection is done on FL's (Forward List), while commands like rollback and obliterate naturally work backwards in time on RL. Currently, an RL is inverted and then given to the patch selection code, which is not convenient at all! Moreover, the actual representation of history of a Darcs repository is (close to being) an RL. So it seems like a proper fix for the bug is to generalize the patch selection code to also work on RL's; which may involve a good amount of typeclass'ing in the relevant modules. I think this will be too big/risky to port to the 2.10 branch, so it will wait for Darcs 2.12.

Ganesh's new not-yet-officially-named stash command

Dan's den

Dan tells more about his work and this sprint in his latest blog post.

A better website and documentation

As a newcomer to the project, Vinh took a look at the documentation, especially the website of the project. He implemented changes to make the front page less intimidating and more organized. He also had a fresh look at our "quickstart" and proposed improvements which we felt were much needed!

Florent's projects

For this sprint, Florent was more an external visitor than a Darcs hacker. He talked about one of his current projects: Pijul, a version control system with another approach. Check out their website!

Conclusion and the next sprint

In the end this sprint turned out to be more productive and crowded than we initially thought! It has been a lot of time since the previous one, so we had a lot of things to share at first. Sprints do make synchronization between contributors more effective. They are also a moment when we can get more concentrated on the Darcs codebase, and spend more time tacking some issue.

Avenue d'Italie, Paris

We would like to thank the IRILL people for hosting the sprint for the third time and our generous donators to make travelling to sprints easier.

We already have a time and a place for the next sprint: Sevilla, Spain in January 2016! The exact moment will be announced later, but you can already start organizing yourself and tell us if you're going.

Thomas, Eric and Ganesh

From left to right: Vinh, Florent, Dan, Ganesh and Eric

This post is intended to be a short summary of my SoC project, as well as my recent trip to Darcs sprint.

Introduction

I am finishing up this post on the train back from the Autumn 2015 Darcs sprint. Today (Sept 20, Sun) was a very fun day full of darcs chatting and coding. By the end of the day we’ve heard a number of presentations

• Ganesh described his work on "stash" command for darcs (naming subject to change!). It involves some refactoring of the rebase code. I hope we would hear more from him on that, because the internal workings are actually quite interesting — I believe it’s the first time singleton types and DataKinds are used in the darcs codebase;
• Florent Becker gave a presentation about Pijul and the theory behind it — A Categorical Theory of Patches by Samuel Mimram and Cinzia Di Giusto, see arXiv:1311.3903;
• Vinh Dang talked about his improvements on the darcs wiki (it’s about time to organize the website), his goal was to make it more accessible to the newcomers;
• Yours truly gave a small presentation, outline of which you will find below:

Looking back

I have spent this summer hacking on DarcsDen as part of the Google Summer of Code program.

My basic goal was to create a "local" version of darcsden. It was not a trivial task to install darcsden (and probably installation is still not very easy!). It uses a third-party software like Redis and CouchDB. During my coding process I modifed darcsden such that it now can be a good choice for local (or lightweight single user) darcs UI. The local darcsden version can be used without any databases, tracking the repositories in the local file system. This way darcsden can be used by a developer on her local computer, like darcsum, (for working with/comparing repositories) as well as a replacement for darcsweb/cgit — a single user web front for darcs repositories.

Besides that a user of a local version can use darcsden’s interactive UI for recording new patches, as well as a command-line tool den for a quick way of browsing the repositories.

Installing darcsden-local is currently not as easy as I want to it be, but I hope that soon you will be able to install it just by running cabal install darcsden or brew install darcsden. As for now, one could do the following:

1. darcs get --lazy http://hub.darcs.net/co-dan/darcsden-local
2. cabal install . or stack install

This should install the darcsden binary and all the related css/js files. You can start darcsden by running darcsden --local. If you open your web browser you should see a list of repositories in the current directory.

However, you might have repositories scattered all over the place, and scanning your whole system for darcs repositories is just inefficient. For this purposes darcs keeps a list of repositories in a file inside your ~/.darcs directory. You can manage that list either by hand, or using the command-line den tool:

• den $PATH — add $PATH to the list of repositories in ~/.darcs/darcsden_repos (if it’s not already present there), start darcsden server in the background and launch the browser pointing to $PATH;
• den — the same as den .;
• den --add $PATH — add $PATH to the list of repositories in ~/.darcs/darcsden_repos;
• den --remove $PATH — remove $PATH from the list of repositories in ~/.darcs/darcsden_repos.

In order to further customize darcsden, one can tweak the configuration file located at ~/.darcs/darcsden.conf. Apart from the usual darcsden settings one may pay attention to the following variables:

• homeDir (default .), points to the "root" directory with repositories. If the list file ~/.darcs/darcsden_repos is not present darcsden will recursively search repositories in that directory
• unLocal, pwLocal: the username and the password of the "local" user

The user/password credentials are required for editing the repositories and recording new patches. However, the den binary should automatically pick them up from the config file and log you in.

Once you are logged in, and you have unrecorded changes in the repository, you can use darcsden UI to record a new patch.

Below you can see an example of recording and merging patches from a branch.

Darsden allows you to create forks/branches of your repositories, and it keeps track of the patch dependencies in your branches.

More "internal" changes:

• Instead of having to specify some parts of the configuration in DarcsDen.Settings, darcsden now uses runtime flags: –hub for using hub-specific modifications, –local for using the local backend and no flag for default behaviour
• The flag actually choose what are called instances — something that a bit less fine-grained than settings. Instances allow you to pick backend, overwrite settings, modify the looks of the front page.
• HTTP-testing using wreq. The previous test suite used selenium and it got bit-rotten. The wreq-based is easier to run and perhaps slightly easier to maintain.
• HTTP auth, which is used as part of the local instance; the den tool utilizes it to log the user in automatically.
• Support for repositories inside directories and nested repositories.
• All the backend code that is used for handling repositories and meta-data on the file system.
• Functionality for downloading zipped dist archives of darcs repositories.
• Assorted mini-fixes

What now?

During the sprint I hacked together some code for viewing suspended patches along the regular ones. The next step would be to have a similar interface for managing the suspended patches.

We have also discussed the possibility of adding rewrite rules implementing short-cut fusion for the directed types in Darcs. In order to see if it’s really worth it we would have to bring back to life the benchmarking suite (or at least check on it!).

It was a really exciting weekend for me and I was delighted to meet some of my IRC friends. As it turns out, it is a small world and despite being from different parts of it we have a bunch of common IRL friends, professors. As the French would (probably not) say, très bien. The next darcs sprint will probably be in January, and probably in Europe, again.

Tagged: darcs, haskell

I am editing this post from IRILL, where the darcs sprint is taking place

One of the things that I particularly like about the Darcs codebase is that you can see that the developers were not shy about using intermediate-slash-advanced Haskell/GHC features to help achieving type safety. You can see GADTs, associated types, phantom types, existential types actively used.

In this post I would like to discuss the representation of patches and the use of type witnesses in darcs.

A word about patches and contexts

This post is intended for people interested in patch theory and its implementation in darcs. A passing familiarity with patches, contexts, inverses, etc is useful, but not required, as we restate some basic definitions in this section.

A primitive patch is a basic unit in patch theory. It constitutes an atomic change in the repository. The definition of a primitive patch may vary, but usually the following changes are considered primitive patches:

• Removing a file from the repository
• Adding file to the repository
• Changing a line n in a file in the repository
• Identity patch, i.e. an empty/non-existing change that does not modify the state of the repository at all

Every primitive patch has a pre-context and a post-context. Roughly, you can think of a pre-context as the full state of the repository before the change was made, and of the post-context as the full state of the repository after the change was applied. We write (x)-A->(y) for a patch A with a pre-context x and a post-context y.

If a primitive patch A has a pre-context a, a post-context o, and a primitive patch B has a pre-context o, a post-context b, then we can combine two patches to obtain a sequential patch AB with the pre-context a and the post-context b.

Every primitive patch (x)-A->(y) has an inverse (y)-A^{-1}->(x), such that (x)-A->(y)-A^{-1}->(x) is equivalent to the identity patch (x)-1->(x).

In the next sections we will see how those things are implemented in darcs.

Primitive patches and witnesses

A primitive patch, which constitutes a single fine grained change, can be represented as a (G)ADT:

data Prim where
    Move :: FileName -> FileName -> Prim
    RmFile :: FileName -> Prim
    AddFile :: FileName -> Prim
    Hunk :: FileName -> Int -> ByteString -> ByteString -> Prim
    RmDir :: FileName -> Prim
    …

We can represent complex patches as sequences of primitive patches:

data Patch where
    NilP :: Patch
    PrimP :: Prim -> Patch
    SeqP :: Patch -> Patch -> Patch

This seems reasonable enough. But if we implement our patch theory this way we seem to be missing something — patches have (pre- and post-) contexts. Having contexs allows us to enforce patch composition on the level of type system. Consider the following definition, which uses phantom types as type witnesses for contexts.

data Prim x y where
    Move :: FileName -> FileName -> Prim x y
    RmFile :: FileName -> Prim x y
    AddFile :: FileName -> Prim x y
    Hunk :: FileName -> Int -> ByteString -> ByteString -> Prim x y
    RmDir :: FileName -> Prim x y
    …

data Patch x y where
    NilP :: Patch x x
    PrimP :: Prim x y -> Patch x y
    SeqP :: Patch x y -> Patch y z -> Patch x z

We call the types with witnesses representing pre- and post-contexts directed types. Intuitively, the directed type D x y has a “direction” from x to y, written as (x)->(y). The Prim datatype looks pretty much like the type actually used in Darcs. The Patch datatype, however, is completely artificial. We will see in the next sections how Darcs really models complex patches.

Directed programming

FL & RL

Two particularly useful directed types used in darcs are directed lists: forwards lists of the type FL a x y and reverse lists RL a x y. Forward lists “go” from the head to the tail; reverse lists “go” from the tail to the head. The lists are polymorphic over a just like regular lists.

data FL a x y where
    NilFL :: FL a x x    {- The empty list stays at (x) -}
    (:>:) :: a x y -> FL a y z -> FL a x z
infixr 5 :>:

For myself, I visualise forward lists like this:

(x) —a—> (y) —b—> (z) ——Nil——> (z)
a :: Patch x y
b :: Patch y z
(a :>: b :>: NilFL) :: FL Patch x z

The reverse lists are “going” from tail to head¹

data RL a x y where
    NilRL :: RL a x x
    (:<:) :: RL a x y -> a y z -> RL a x z
infixl 5 :<:

(Mnemonically, the the head is always “greater” than the tail)

The reason I used the word “go” inside quotation marks so far is the following. Reverse lists and forward lists represent the same concept: a sequence of patches (or a sequence of directed things for that matter). They only differ in the associativity of the elements. Forward lists associate to the right, but reverse lists associate to the left.

p = Move "foo" "bar"

-- [[(x) --Nil--> (x) -p-> (y)] -id-> (y)] -p^{-1}-> (x)
example :: RL Patch x x
example = NilRL :<: p :<: Identity :<: inverse p

-- (x) -p-> [(y) -p^{-1}-> (x) --Nil--> (x)]
example2 :: FL Patch x x
example2 = p :>: inverse p :>: NilFL

The right-associated/reverse lists provide easy access to the last element of the sequence; the left-associated/forward lists provide easy access to the first element of the sequence. Therefore, if we view a repository as a directed sequence of patches, right-associated lists are useful for operations that work on the “latest” patches in the repository (such as record/unrecord), and left-associated lists are useful for commands that scan the repository from the beginning (like clone).

We can reassociate the lists easily, and verify that the two representations are isomoprhic:²

reverseRL :: RL a wX wY -> FL a wX wY
reverseRL = r NilFL
-- the type signature of r basically gives us an invariant
-- wZ is slowly "decreasing" reaching the point where
-- wZ = wX; at that point the first argument is of type FL a wX wY
  where r :: FL a wZ wY -> RL a wX wZ -> FL a wX wY
        r a NilRL = a
        r a (xs :<: x) = r (x :>: a) xs

For example,

-- [[(x) --Nil--> (x) -p-> (y1)] -q-> (y2)] -r-> (z)

Turns into

-- (x) -p-> [(y1) -q-> (y2) -r-> [(z) --Nil--> (z)]]

Exercise: write a function reverseFL :: FL a wX wY -> RL a wX wY

We can write a lot of directed analogues of familiar list functions. For example, here is a directed append:

infixl 5 +<+
(+<+) :: RL a wX wY -> RL a wY wZ -> RL a wX wZ
xs +<+ NilRL = xs
xs +<+ (ys :<: y) = (xs +<+ ys) :<: y

Exercise: write directed append for forward lists: (+>+) :: FL a wX wY -> FL a wY wZ -> FL a wX wZ

Type witnesses

So we wrote a bunch of standard list functions for directed lists; what about some of the other functions? Can we, for example, implement filter for directed lists. Can it look like filterFL :: (a wX wY -> Bool) -> FL a wX wY? Well, we can try writing

filterFL :: (a wX wY -> Bool) -> FL a wX wY
filterFL p NilFL = NilFL
filterFL p (x :>: xs) | p x = filterFL p xs
                      | otherwise = x :>: filterFL p xs

However, under closer scrutiny we realize that it does not typecheck! In the second clause of filterFL we have the following information:

x :: a x y, xs :: FL a y z
filterFL xs :: FL a y z

Thus, in the first case (in which p x holds) we try to return something of the type FL a wY wZ, when FL a wX wZ was expected. It is clear that generally we can do this only if x :: a wX wX, i.e. wY = wX. But a simple predicate of the type p :: a wX wZ -> Bool won’t tell us anything about that. We need an additional type witness in our system telling us that if p x holds, then x :: a wX wX. For that purpose we introdue the EqCheck datatype.

data EqCheck wX wY where
    IsEq :: EqCheck wX wX
    NotEq :: EqCheck wX wY

then the type of a predicate would be

type Pred a = forall wX wY. a wX wY -> EqCheck wX wY

If (p x) = IsEq, then the typechecker will know that x :: a wX wX. We can then finally write

filterFL :: Pred a -> FL a wX wY -> FL a wX wY
filterFL p NilFL = NilFL
filterFL p (x :>: xs) | IsEq <- p x = filterFL p xs
                      | otherwise   = x :>: filterFL p xs

EqCheck is used this way in the darcs source code to e.g., filter our internal patches. Sometimes darcs stores information — like suspended patches — in the so called internal patches. Every patch type implements the internal patch checker (code slightly adapted):

-- |Provides a hook for flagging whether a patch is "internal" to the repo
-- and therefore shouldn't be referred to externally, e.g. by inclusion in tags.
-- Note that despite the name, every patch type has to implement it, but for
-- normal (non-internal) types the default implementation is fine.
-- Currently only used for rebase internal patches.
class MaybeInternal p where
    -- | @maybe (const NotEq) (fmap isInternal patchInternalChecker) p@
    -- returns 'IsEq' if @p@ is internal, and 'NotEq' otherwise.
    -- The two-level structure is purely for efficiency: 'Nothing' and 'Just (InternalChecker (const NotEq))' are
    -- semantically identical, but 'Nothing' allows clients to avoid traversing an entire list.
    -- The patch type is passed as an 'FL' because that's how the internals of named patches are stored.
    patchInternalChecker :: Maybe (forall wX wY . FL p wX wY -> EqCheck wX wY)
    patchInternalChecker = Nothing

When the user runs darcs tag in the repository, darcs creates a dummy patch that explicitly depends on all the previous patches — apart from the internal ones of course. Thus, the tag command uses the following function (slightly adapted):

filterNonInternal :: MaybeInternal p => PatchSet p wX wY -> PatchSet p wX wY
filterNonInternal =
  case patchInternalChecker of
     Nothing -> id
     Just f -> \l -> PatchSet (filterRL (f . patchcontents . hopefully) (unPatchSet l))

where a PatchSet is the list of PatchInfoAnd patches — patches together with the meta-information.

It is worth noting that EqCheck x y is isomorphic Maybe (x :~: y), but the propositional equality datatype has only been added to base since 4.7.0.0. In the future versions darcs will probably switch to using Data.Type.Equality.

Conclusion

We’ve briefly touched upon patch representation in darcs and talked about directed types and directed programming.

A good if a bit outdated reference is Jason Dagit’s master thesis (specifically the bits from chapter 4). The wiki is currently lacking in material, but I hope to improve the situation eventually.

Next time we will probably discuss either directed pairs and their use in darcs, or sealed datatypes, or both.

Tagged: darcs, haskell

News and discussions

1. The next Darcs Sprint will take place in Paris on September 18-20th. Please add yourself to the wiki page if you're going!
   
   • http://lists.osuosl.org/pipermail/darcs-users/2015-August/027134.html
   • http://darcs.net/Sprints/2015-09
2. Darcs 2.10.1 has been released (bugfixes, dependency versions bump):
   
   • http://lists.osuosl.org/pipermail/darcs-users/2015-July/027130.html

Issues resolved (19)

issue2102 Guillaume Hoffmann

• document packs in help of clone and optimize http
• http://bugs.darcs.net/issue2102

issue2307 Daniil Frumin

• Add information about 'darcs help manpage' and 'darcs help markdown'
• http://bugs.darcs.net/issue2307

issue2308 Ben Franksen

• use command agnostic comment for changes summary
• http://bugs.darcs.net/issue2308

issue2327 Alain91

• ask user to keep trying if locking fails
• http://bugs.darcs.net/issue2327

issue2420 Ben Franksen

• commandGetArgPossibilities for apply should not list all files in the repo
• http://bugs.darcs.net/issue2420

issue2421 Guillaume Hoffmann

• don't fail if caches are not writeable
• http://bugs.darcs.net/issue2421

issue2423 Alain91

• fix diff only respecting --diff-command when a diff.exe is present
• http://bugs.darcs.net/issue2423

issue2433 Guillaume Hoffmann

• fix help strings of send and fetch
• http://bugs.darcs.net/issue2433

issue2438 Guillaume Hoffmann

• no longer build patch index by default on cloning
• http://bugs.darcs.net/issue2438

issue2444 Ben Franksen

• added default interactivity parameter to isInteractive
• http://bugs.darcs.net/issue2444

issue2446 Guillaume Hoffmann

• in fast-export delete dirs before dumping contents
• http://bugs.darcs.net/issue2446

issue2447 Ben Franksen

• by using getNonrangeMatch as in ShowFiles
• http://bugs.darcs.net/issue2447

issue2448 Gian Piero Carrubba

• enable -h/--hash flag for the diff command
• http://bugs.darcs.net/issue2448

issue2449 Ganesh Sittampalam

• use new hook in shelly to handle test output as binary
• http://bugs.darcs.net/issue2449

issue2451 Ben Franksen

• allow terminfo 0.4.*
• http://bugs.darcs.net/issue2451

issue2457 Ben Franksen

• fix darcs-test command line options
• http://bugs.darcs.net/issue2457

issue2461 Ben Franksen

• this time also when no file names are specified
• http://bugs.darcs.net/issue2461

issue2461 Ben Franksen

• darcs log --repo=remoterepo creates and populates _darcs
• http://bugs.darcs.net/issue2461

issue2463 Joachim Breitner

• building darcs on powerpc
• http://bugs.darcs.net/issue2463

Patches applied (145)

See darcs wiki entry for details.

Darcs is a patch-centric version control system. In Darcs, there is no “correct” linear history of a repository – rather, there is a poset of patches. That means that most of the time you are pushing and pulling changes you can cherry-pick patches without a problem. However, in some cases you cannot perform a pull (or some other operation on the repository) smoothly. Sometimes it is necessary to rewrite the “history” – i.e. modify a patch that is a dependency of one or more other patches. For those cases darcs rebase comes in handy. To put it in the words of the implementor “Rebase is a workaround for cases where commutation doesn’t do enough”.

A repository can change it’s state from rebase-in-progress back to normal if there are no suspended patches left. However, be aware that you cannot unsuspend a patch¹ if you have unrecorded changes in the repository. In light of this, I suggest recording a temporary patch with current changes

darcs record -am DRAFT

You can suspend that patch at the beginning of your rebase process and apply it at the end.

General overview of rebase

darcs rebase is an operation (or, rather, a family of operations) that allows one to make changes “deep down” in the repository history. One of the crucial things that allows for rebase to work is the fact that since darcs 2.10 patches can be suspended. When one performs any of the darcs rebase commands, the repository moves to a special rebase-in-progress state. In this state repository contains a pristine, a set of patches, a working copy, and — in addition to all the usual stuff — a set of suspended patches. Suspended patches are not active in the repository — that is, they are not applied.

Let’s go over the rebase subcommands

rebase log/rebase changes

This is simple: list the suspended patches

rebase suspend

Moves selected patches into the suspended state. Once the patch is suspended it is no longer active in the repository.

Note: once you suspend a patch, it changes its identity. That means that even if you suspend a patch and unsuspend it immediately, you will get a different repository that you have started with. Let this be a good reason (one of a couple!) for doing rebase on a separate branch.

> cat file
test
123
> darcs rebase suspend
	patch 64523bc4622fad02a4bdb9261887628b7997ebdd
Author: Daniil Frumin 
Date:   Thu Jul 23 18:49:30 MSK 2015
  * 123
Shall I suspend this patch? (1/5)  [ynW…], or ? for more options: y
patch cc54d7cf4b9e3d13a24ce0b1b77b76581d98d75d
Author: Daniil Frumin 
Date:   Thu Jul 23 18:43:53 MSK 2015
  * Test
Shall I suspend this patch? (2/5)  [ynW…], or ? for more options: d
Rebase in progress: 1 suspended patches
> darcs rebase log
patch 64523bc4622fad02a4bdb9261887628b7997ebdd
Author: Daniil Frumin 
Date:   Thu Jul 23 18:49:30 MSK 2015
  * 123
Shall I view this patch? (1/?) [yN…], or ? for more options: y
[123
Daniil Frumin **20150723154930
 Ignore-this: 43e09e6503ac74688e74441dc29bce25
] hunk ./file 2
+123
Rebase in progress: 1 suspended patches
> cat file
test

rebase unsuspend

Does the opposite of suspend: applies a suspended patch to the repository and changes its state to normal.

> darcs rebase unsuspend                                                                                       
patch 64523bc4622fad02a4bdb9261887628b7997ebdd
Author: Daniil Frumin 
Date:   Thu Jul 23 18:49:30 MSK 2015
  * 123
Shall I unsuspend this patch? (1/1)  [ynW…], or ? for more options: y
Do you want to unsuspend these patches? [Yglqk…], or ? for more options: y
Rebase finished!

rebase apply

Rebase apply takes a patch bundle and tries to apply all the patches in the bundle to the current repository. If a patch from the bundle conflicts with a local patch, then the local patch gets suspended. You will thus have a chance to resolve the conflict by amending your conflicting patches, at a price of.. well, changing the identity of your local patches.

rebase pull

Sort of like rebase apply, but instead of a patch bundle it obtains the patches from a remote repository.

Specifically, rebase pull applies all the remote patches, one-by-one, suspending any local patches that conflict. We will see more of rebase pull in the second example.

Example 1: suspending local changes

Imagine the following situation: at point A you add a configuration file to your repository, then you record a patch B that updates the settings in the configuration file. After that you make some more records before you realize that you’ve included by accident your private password in patch A! You want to get rid of it in your entire history, but you can’t just unrecord A, because B depends on A, and possibly some other patches depend on B.

The contents of the configuration file after patch A:

port = 6667
host = irc.freenode.net
password = awesomevcs 

Patch B, diff:

@@ -1,3 +1,4 @@
-port = 6667
+port = 6697
+usessl = True
host = irc.freenode.net
password = awesomevcs

You cannot just amend patch A, because the patch B depends on A:

> darcs amend                                                                                                         
patch 1925d640f1f3180cb5b9e64260c1b5f374fce4ca
Author: Daniil Frumin 
Date:   Tue Jul 21 13:23:07 MSK 2015
  * B

Shall I amend this patch? [yNjk…], or ? for more options: n

Skipping depended-upon patch:
patch 22d7c8da83141f8b1f80bdd3eff02064d4f45c6b
Author: Daniil Frumin 
Date:   Tue Jul 21 13:22:24 MSK 2015
  * A

Cancelling amend since no patch was selected.

What we will have to do is temporarily suspend patch B, amend patch A, and then unsuspend B.

> darcs rebase suspend
patch 1925d640f1f3180cb5b9e64260c1b5f374fce4ca
Author: Daniil Frumin 
Date:   Tue Jul 21 13:23:07 MSK 2015
  * B

Shall I suspend this patch? (1/2)  [ynW…], or ? for more options: y
patch 22d7c8da83141f8b1f80bdd3eff02064d4f45c6b
Author: Daniil Frumin 
Date:   Tue Jul 21 13:22:24 MSK 2015
  * A

Shall I suspend this patch? (2/2)  [ynW…], or ? for more options: d
Rebase in progress: 1 suspended patches

At this point, the state of our repository is the following: there is one (active) patch A, and one suspended patch B.

> darcs rebase changes -a
patch 4c5d45230dc146932b21964aea938e2a978523eb
Author: Daniil Frumin 
Date:   Tue Jul 21 13:28:58 MSK 2015
  * B

Rebase in progress: 1 suspended patches
> darcs changes -a
patch 21f56dfb425e4c49787bae5db4f8869e96787fb2
Author: Daniil Frumin 
Date:   Tue Jul 21 13:28:49 MSK 2015
  * A

Rebase in progress: 1 suspended patches
> cat config
port = 6667
host = irc.freenode.net
password = awesomevcs
> $EDITOR config # remove the password bit
> darcs amend
patch 22d7c8da83141f8b1f80bdd3eff02064d4f45c6b
Author: Daniil Frumin 
Date:   Tue Jul 21 13:22:24 MSK 2015
  * A

Shall I amend this patch? [yNjk…], or ? for more options: y
hunk ./config 3
-password = awesomevcs
Shall I record this change? (1/1)  [ynW…], or ? for more options: y
Do you want to record these changes? [Yglqk…], or ? for more options: y
Finished amending patch:
patch 21f56dfb425e4c49787bae5db4f8869e96787fb2
Author: Daniil Frumin 
Date:   Tue Jul 21 13:28:49 MSK 2015
  * A

Rebase in progress: 1 suspended patches

Now that we’ve removed the password from the history, we can safely unsuspend patch B (in this particular situation we actually know that applying B to the current state of the repository won’t be a problem, because B does not conflict with our modified A)

> darcs rebase unsuspend
patch 1925d640f1f3180cb5b9e64260c1b5f374fce4ca
Author: Daniil Frumin 
Date:   Tue Jul 21 13:23:07 MSK 2015
  * B

Shall I unsuspend this patch? (1/1)  [ynW…], or ? for more  options: y
Do you want to unsuspend these patches? [Yglqk…], or ? for more options: y
Rebase finished!

And that’s done!

> cat config
port = 6697
usessl = True
host = irc.freenode.net

You may use this rebase strategy for removing sensitive information from the repository, for removing that 1GB binary .iso that you added to your repository by accident, or for combining two patches into one deep down in the patchset.

Example 2: developing against a changing upstream – rebase pull

Imagine you have a fork R’ of a repository R that you are working on. You are implementing a feature that involves a couple of commits. During your work you record a commit L1 that refractors some common datum from modules A.hs and B.hs. You proceed with your work recording a patch L2. At this point you realise that after you forked R, the upstream recorded two more patches U1 and U2, messing with the common datum in A.hs. If you just pull U1 into your fork R’, you will have a conflict, that you will have to resolve by recording another patch on top.

      S
     / \
    /   \
   L1   U1

Note: if you run darcs rebase pull in R’, then the only patches that will be suspended are the ones which are already in R’. Because suspended patches gain new identity, make sure that you do not have other people’s conflicting patches present in R’.

The way to solve this would be to first do darcs rebase pull, which would suspend the conflicting patches, and then start unsuspending the patches one by one, making sure that you fix any conflicts that may arise after each unsuspend.

Consider a concrete example with two repositories rep1 and rep1_0.

rep1_0 > darcs changes 
patch ebaccd5c36667b7e3ee6a49d25ef262f0c7edf2b
Author: Daniil Frumin 
Date:   Mon Jul 27 20:56:25 MSK 2015
  * commit2

patch a7e0d92a53b0523d0224ef8ffae4362adf854485
Author: Daniil Frumin 
Date:   Mon Jul 27 20:56:25 MSK 2015
  * commit1
	rep1_0 > darcs diff —from-patch=commit2
patch ebaccd5c36667b7e3ee6a49d25ef262f0c7edf2b
Author: Daniil Frumin 
Date:   Mon Jul 27 20:56:25 MSK 2015
  * commit2
diff -rN -u old-rep1_0/dir1/file2 new-rep1_0/dir1/file2
— old-rep1_0/dir1/file2	1970-01-01 03:00:00.000000000 +0300
    +++ new-rep1_0/dir1/file2	2015-07-28 12:25:54.000000000 +0300
@@ -0,0 +1 @@
+double whatsup
	rep1_0 > cd ../rep1
rep1 > darcs changes
patch e3df0e23a3915910a81eb8181d7b3669e8f270a9
Author: Daniil Frumin 
Date:   Tue Jul 28 12:27:55 MSK 2015
  * commit2’

patch a7e0d92a53b0523d0224ef8ffae4362adf854485
Author: Daniil Frumin 
Date:   Mon Jul 27 20:56:25 MSK 2015
  * commit1
rep1 > darcs diff —from-patch=“commit2’”
patch e3df0e23a3915910a81eb8181d7b3669e8f270a9
Author: Daniil Frumin 
Date:   Tue Jul 28 12:27:55 MSK 2015
  * commit2’
diff -rN -u old-rep1/dir1/file2 new-rep1/dir1/file2
— old-rep1/dir1/file2	1970-01-01 03:00:00.000000000 +0300
+++ new-rep1/dir1/file2	2015-07-28 12:28:39.000000000 +0300
@@ -0,0 +1 @@
+touch file2
\ No newline at end of file
diff -rN -u old-rep1/file1 new-rep1/file1 — old-rep1/file1	2015-07-28 12:28:39.000000000 +0300 +++ new-rep1/file1	2015-07-28 12:28:39.000000000 +0300
@@ -1 +1 @@
-whatsup
\ No newline at end of file
+double whatsup
\ No newline at end of file

The patch commit2 from rep1_0 conflicts with commit2’ from rep1.

rep1 > darcs rebase pull ../rep1_0
patch ebaccd5c36667b7e3ee6a49d25ef262f0c7edf2b
Author: Daniil Frumin 
Date:   Mon Jul 27 20:56:25 MSK 2015
  * commit2
Shall I pull this patch? (1/1)  [ynW…], or ? for more options: y
Do you want to pull these patches? [Yglqk…], or ? for more options: y
The following local patches are in conflict:
patch e3df0e23a3915910a81eb8181d7b3669e8f270a9
Author: Daniil Frumin 
Date:   Tue Jul 28 12:27:55 MSK 2015
  * commit2’
Shall I suspend this patch? (1/1)  [ynW…], or ? for more options: y
Do you want to suspend these patches? [Yglqk…], or ? for more options: y
Finished pulling.
Rebase in progress: 1 suspended patches

Now we have one patch — commit2’ — in the suspended state. We want to resolve the conflict by amending commit2’. We will do that by unsuspending it and manually editing out the conflicting lines. This will also make it depend on commit2.

rep1 > darcs rebase unsuspend
patch e3df0e23a3915910a81eb8181d7b3669e8f270a9
Author: Daniil Frumin 
Date:   Tue Jul 28 12:27:55 MSK 2015
  * commit2’
Shall I unsuspend this patch? (1/1)  [ynW…], or ? for more options: y
Do you want to unsuspend these patches? [Yglqk…], or ? for more options: d
We have conflicts in the following files:
./dir1/file2

Rebase finished!
rep1 > cat dir1/file2
v v v v v v v

=============
double whatsup
*************
touch file2
^ ^ ^ ^ ^ ^ ^
rep1 > $EDITOR dir1/file2
rep1 > darcs amend -a
patch 40b3b4123c78dba6a6797feb619572072654a9cd
Author: Daniil Frumin 
Date:   Tue Jul 28 12:32:56 MSK 2015
  * commit2’
Shall I amend this patch? [yNjk…], or ? for more options: y
Finished amending patch:
patch c35867259f187c1bc30310f1cacb34c1bb2cce41
Author: Daniil Frumin 
Date:   Tue Jul 28 12:34:30 MSK 2015
  * commit2’
rep1 > darcs mark-conflicts
No conflicts to mark.

Another repository saved from conflicting patches, yay!

Tagged: darcs, haskell, rebase

Darcs 2.10.1 has been released!

Citing the official release notes

> The darcs team is pleased to announce the release of darcs 2.10.1 !
> ..
> 
> # What's new in 2.10.1 (since 2.10.0) #
> 
> - generalized doFastZip for darcsden support
> - support terminfo 0.4, network 2.6, zlib 0.6, quickcheck 2.8 and
> attoparsec 0.13
> - errorDoc now prints a stack trace (if profiling was enabled) (Ben Franksen)
> - beautified error messages for command line and default files (Ben Franksen)
> - fixed the following bugs:
>       - issue2449: test harness/shelly: need to handle
> mis-encoded/binary data (Ganesh Sittampalam)
>       - issue2423: diff only respecting --diff-command when a diff.exe
> is present (Alain91)
>       - issue2447: get contents of deleted file (Ben Franksen)
>       - issue2307: add information about 'darcs help manpage' and
> 'darcs help markdown' (Dan Frumin)
>       - issue2461: darcs log --repo=remoterepo creates and populates
> _darcs (Ben Franksen)
>       - issue2459: cloning remote repo fails to use packs if cache is
> on a different partition (Ben Franksen)
> 
> # Feedback #
> 
> If you have an issue with darcs 2.10.0, you can report it on
> http://bugs.darcs.net/ . You can also report bugs by email to
> bugs at darcs.net, or come to #darcs on irc.freenode.net.

I’ve updated Mac OS to version 2.10.1. You can install it with

brew install http://darcs.covariant.me/darcs.rb

Tagged: darcs, haskell, homebrew

Recently, I’ve implemented HTTP Basic auth for darcsden and wrote a simple wreq test for it. In this post I would like to outline the main technical details.

Server side

Transient storage

A lot of darcsden code is (especially the parts that are closer to the users’ web browser — handlers, pages, so on) is written around sessions. Sessions are stored in a special storage — implemented by the DarcsDen.Backend.Transient, but if we abstract away from the details we have a Session datatype. Authorization and authentication information is handled by sessions using functions setUser :: (BTIO bt) => Maybe User -> Session -> Snap Session, notice :: (BTIO bt) => String -> Session -> Snap () (display a message to the user) and others. The BTIO bt part is just a synonym for

type BTIO bt = (BackendTransient bt, ?backendTransient :: bt, MonadIO (BackendTransientM bt))

Which basically says that we are operating with a transient backend that supports all of necessary operations, and we can also do IO in it. Right now there are only two transient backends (read: two ways of storing sessions): Redis and in-process memory.

Running sessions

If we have a piece of Snap code that we want to “sessionify” we use the following interface:

withSession :: (BTIO bt) => (Session -> Snap ()) -> Snap ()

What this does is it basically checks for a cookie — in case it is present it grabs the session information from the storage (in accordance with the cookie); if the cookie is not present it creates a new session and stores it in a cookie.

If we have a page of a type Session -> Snap (), we might want to give user an option to do HTTP authentication on that page. We introduce another function

withBasicAuth :: (BP bp, BTIO bt) 
              => (Session -> Snap ()) 
              -> (Session -> Snap ())
withBasicAuth act s = do
    rq  do
          rawHeader <- maybe throwChallenge return $ getHeader “Authorization” rq
          let (Just (usr,pw)) = getCredentials rawHeader
          c  errorPage “Unknown user”
            Just u -> if checkPassword (fromBS pw) u
                         then doLogin (fromBS usr)
                         else errorPage “Bad password”
      _ -> act s

So, what is going on in here? First of all, we check if the “login” parameter is set to “true”. If it does, we try to get the “Authorization” header, de-encode it, and check whether the credentials are good.

throwChallenge :: Snap a
throwChallenge = do
    modifyResponse $ (setResponseStatus 401 “Unauthorized”) .
                     (setHeader “WWW-Authenticate” “Basic realm=darcsdenrealm”)
    getResponse >>= finishWith

If the response header is present, it is of a form Basic x, where x is a base64 encoded string user:password. We can extract the credentials from the header like this:

import qualified Data.ByteString          as B
import qualified Data.ByteString.Base64   as B
…
getCredentials :: B.ByteString  — ^ Header
               -> Maybe (B.ByteString, B.ByteString) — ^ Possibly (username, password)
getCredentials header =
    if (isInfixOf “Basic “ header)
      then fmap extract (hush (B.decode (B.drop 6 header)))
      else Nothing
  where
    extract cred = case (B.breakByte (c2w ‘:’) cred) of
                     (usr, pw) -> (usr, safeTail pw)

On the client

The tests that I am currently writing for darcsden are all of the same form: I use wreq to do requests to the darcsden server, then, if necessary, I run taggy to extract information from the webpage, and compare it to the “canonical” information.

As it turns out, doing HTTP Basic Auth is very easy with wreq! First of all, we define a function for doing a GET request that will do some exception handling for us:

getSafeOpts :: Options -> String -> IO (Either Status (Response ByteString))
getSafeOpts opts url = fmap Right (getWith opts url) `catch` hndlr
  where
    hndlr (StatusCodeException s _ _) = return (Left s)
    hndlr e = throwIO e

This way, we won’t get a runtime exception when accessing a non-existing page or getting a server error. Doing a GET request with HTTP Basic Auth is now very easy:

getWithAuth :: (String, String) -> String -> IO (Either Status (Response ByteString))
getWithAuth (username,pw) url = getSafeOpts opts url
  where
    opts = defaults & auth ?~ basicAuth (toBS username) (toBS pw)
                    & param “login” .~ [“true”]

In that snippet we use lenses to set up an auth header and an HTTP parameter (?login=true).

Finally, after obtaining a Response ByteString, we can parse it with taggy-lens:

parsed :: Fold (Response L.ByteString) Text.Taggy.Node
parsed = responseBody 
       . to (decodeUtf8With lenientDecode)
       . html

We can then play with it in GHCi

*Main> Right r  r ^.. parsed
[NodeElement (Element {eltName = “DOCTYPE”, eltAttrs = fromList [(“html”,””)], eltChildren = [NodeElement (Element {eltName = “html”, eltAttrs = fromList [], eltChildren = [NodeElement (Element {eltName = “head”, eltAttrs = fromList [], eltChildren = [NodeElement (Element {eltName = “title”, eltAttrs = fromList [], eltChildren = [NodeContent “localhost”]}),NodeElement (Element {eltName = “link”, eltAttrs = fromList [(“href”,”http://localhost:8900/public/images/favicon.ico”),…

If we want to check that we are indeed logged in correctly, we should look for the “log out” button. Taggy does all the heavy lifting for us, we just have to write down a lens (more precisely, a fold) to “extract” a logout button from the page

logoutButton :: HasElement e => Fold e Text
logoutButton = allAttributed (ix “class” . only “logout”)
             . allNamed (only “a”)
             . contents

logoutButton searches for <div class=“logout”> and returns the text in the link inside the div. There might be many such links inside the node, hence we use a fold.

*Main> r ^.. parsed . logoutButton
[“log out”]

In this case, since we only care if such link is present, we can use a preview

*Main> r ^? parsed . logoutButton
Just “log out”

Conclusion

Well, that’s about it for now. I regret taking way too much time writing this update, and I hope to deliver another one soon. Meanwhile, some information regarding the darcsden SoC project can be found on the wiki.

Tagged: darcs, haskell

Hi all,

The darcs team is pleased to announce the release of darcs 2.10.1 !

Downloading

The easiest way to install darcs 2.10.1 from source is by first installing the Haskell Platform (http://www.haskell.org/platform). If you have installed the Haskell Platform or cabal-install, you can install this release by doing:

$ cabal update
$ cabal install darcs-2.10.1

Alternatively, you can download the tarball from http://darcs.net/releases/darcs-2.10.1.tar.gz and build it by hand as explained in the README file.

The 2.10 branch is also available as a darcs repository from http://darcs.net/releases/branch-2.10

What's new in darcs 2.10.1 (since darcs 2.10.0)

• support terminfo 0.4, network 2.6, zlib 0.6, quickcheck 2.8 and attoparsec 0.13
• errorDoc now prints a stack trace (if profiling was enabled) (Ben Franksen)
• beautified error messages for command line and default files (Ben Franksen)
• generalized doFastZip for darcsden support
• fixed the following bugs:
• issue2449: test harness/shelly: need to handle mis-encoded/binary data (Ganesh Sittampalam)
• issue2423: diff only respecting --diff-command when a diff.exe is present (Alain91)
• issue2447: get contents of deleted file (Ben Franksen)
• issue2307: add information about 'darcs help manpage' and 'darcs help markdown' (Dan Frumin)
• issue2461: darcs log --repo=remoterepo creates and populates _darcs (Ben Franksen)
• issue2459: cloning remote repo fails to use packs if cache is on a different partition (Ben Franksen)

Feedback

If you have an issue with darcs 2.10.1, you can report it on http://bugs.darcs.net/ . You can also report bugs by email to bugs@darcs.net, or come to #darcs on irc.freenode.net.

Recently I’ve updated my Darcs homebrew build to Darcs 2.10. You can install it with

brew install http://darcs.covariant.me/darcs.rb

The formula contains a context (--exact-version) and it is a static binary.

Tagged: darcs, haskell, homebrew

Introduction

This is my first post from the (hopefuly fruitful!) series of blog posts as part of my Haskell SoC project. I will spend a great chunk of my summer hacking away on DarcsDen; in addition, I will document my hardships and successes here. You can follow my progress on my DarcsHub.

This particular post will be about my working environment.

The problem

Hoogle is an amazing tool that usually needs no introduction. Understandably, the online version at haskell.org indexes only so many packages. This means that if I want to use hoogle to search for functions and values in packages like darcs and darcsden, I will have to set up a local copy.

Cabal sandboxing is a relatively recent feature of the Cabal package manager, but I don’t think it is reasonable in this day to install from the source (let alone develop) a Haskell package without using sandboxing.

The problem seems to be that the mentioned tools do not play well together out of the box, and some amount of magic is required. In this note I sketch the solution, on which I’ve eventually arrived after a couple of tries.

Using hoogle inside a Cabal sandbox

The presumed setup: a user is working on a package X using the cabal sandboxes. The source code is located in the directory X and the path to the cabal sandbox is X/.cabal-sandbox.

Step 1: Install hoogle inside the sandbox. This is simply a matter of running cabal install hoogle inside X. If you want to have a standard database alongside the database for your packages in development, now is the time to do .cabal-sandbox/bin/hoogle data.

Step 2: Generate haddocks for the packages Y,Z you want to use with hoogle. In my case, I wanted to generate haddocks for darcs and darcsden. This is just a matter of running cabal haddock --hoogle in the correct directory.

Step 3: Convert haddocks to .hoo files. Run the following commands in X/:

.cabal-sandbox/bin/hoogle convert /path/to/packageY/dist/doc/html/*/*.txt

You should see something like

Converting /path/to/packageY/dist/doc/html/Y/Y.txt
Converting Y... done

after which the file Y.hoo appears in /path/to/packageY/dist/doc/html/Y/

Step 4: Moving and combining databases. The hoogle database should be stored in .cabal-sandbox/share/*/hoogle-*/databases. Create such directory, if it’s not present already. Then copy the ‘default’ database to that folder:

cp .cabal-sandbox/hoogle/databases/default.hoo .cabal-sandbox/share/*/hoogle-*/databases

Finally, you can combine your Y.hoo with the default database.

.cabal-sandbox/bin/hoogle combine /path/to/packageY/dist/doc/html/*/*.hoo .cabal-sandbox/share/*/hoogle-*/databases/default.hoo
mv default.hoo .cabal-sandbox/share/*/hoogle-*/databases/default.hoo

And you are done! You can test your installation

$ .cabal-sandbox/bin/hoogle rOwner
DarcsDen.State.Repo rOwner :: Simple Lens (Repository bp) String

For additional usability, consider adding .cabal-sandbox/bin to your $PATH.

Tagged: cabal, darcs, haskell, hoogle

I recently learned of a serious undocumented vulnerability in the ssh package. This is a minimal ssh server implementation used by darcsden to support darcs push/pull. If you use the ssh package, or you have darcsden’s darcsden-ssh server running, you should upgrade to/rebuild with the imminent ssh-0.3 release right away. Or if you know of someone like that, please let them know.

darcsden is of course the basis for Darcs Hub. Here’s the announcement I sent to users there a few days ago, with more details.

Hi all,

The darcs team is pleased to announce the release of darcs 2.10.0.

Downloading

The easiest way to install darcs 2.10.0 from source is by first installing the Haskell Platform (http://www.haskell.org/platform). If you have installed the Haskell Platform or cabal-install, you can install this release by doing:

$ cabal update
$ cabal install darcs-2.10.0

Alternatively, you can download the tarball from http://darcs.net/releases/darcs-2.10.0.tar.gz and build it by hand as explained in the README file.

The 2.10 branch is also available as a darcs repository from http://darcs.net/releases/branch-2.10

Feedback

If you have an issue with darcs 2.10.0, you can report it via the web on http://bugs.darcs.net/ . You can also report bugs by email to bugs at darcs.net, or come to #darcs on irc.freenode.net.

What's new since darcs 2.8.5

New features

• darcs rebase: enable deep amending of history (Ganesh Sittampalam)
• darcs pull --reorder: keep local-only patches on top of mainstream patches (Ale Gadea, Ganesh Sittampalam)
• darcs dist --zip: generate a zip archive from a repository (Guillaume Hoffmann)
• patch bundle contexts are minimized by default. Enables bundles to be applied to more repositories. (Guillaume Hoffmann)
• darcs convert export/import for conversion to/from VCSes supporting the fast-export protocol (Petr Rockai, Owen Stephens, Guillaume Hoffmann, Lele Gaifax, Ben Franksen)
• darcs test --backoff: exponential backoff test strategy, faster than bisect on big repositories (Michael Hendricks)
• work normally on sshfs-mounted repositories (Nathaniel Filardo)
• automatic detection of file/directory moves, and of token replaces (Jose Neder)
• patience diff algorithm by default (Jose Neder)
• interactive mode for whatsnew (Dan Frumin)
• tag --ask-deps to create tags that may not include some patches (Ganesh Sittampalam)

User Interface

• add a last question after all patches have been selected to confirm the whole selection (Florent Becker)
• command names:
  • clone is the new name of get and put
  • log is the new name of changes
  • amend is the new name of amend-record
• show output of log into a pager by default (Guillaume Hoffmann)
• the output of log is more similar to git's:
  • show patch hash in UI (hash of the patch's metadata)
  • put author and date on separate lines (Guillaume Hoffmann)
• enable to match on patch hash prefix with -h and --hash (Guillaume Hoffmann, Gian Piero Carrubba)
• better messages:
  • better error messages for http and ssh errors (Ernesto Rodriguez)
  • init, add, remove, move and replace print confirmation messages (Guillaume Hoffmann)
• rollback only happens in the working copy (Florent Becker, Guillaume Hoffmann)
• darcs send no longer tries to send a mail by default (Eric Kow)
• when no patch name given, directly invoke text editor (Jose Neder, Guillaume Hoffmann)
• use nano as default text editor instead of vi (Guillaume Hoffmann)
• keep log files for patch name and mail content in _darcs (Ale Gadea)
• optimize and convert are now supercommands (Guillaume Hoffmann)
• improve darcs help environment and darcs help markdown (Radoslav Dorcik, Guillaume Hoffmann)
• warn about duplicate tags when creating a new one (Ale Gadea)
• allow darcs mv into known, but deleted in working, file (Owen Stephens)
• improve--not-in-remote, allowing multiple repos and use default (Owen Stephens)

Performance

• faster darcs diff (Petr Rockai)
• faster log and annotate thanks to patch index data structure (BSRK Aditya, Benedikt Schmidt, Eric Kow, Guillaume Hoffmann, Ganesh Sittampalam)
• faster push via ssh by using compression (Ben Franksen)
• cloning to an ssh destination (formerly darcs put) is more efficient (Guillaume Hoffmann)
• faster internal representation of patch hashes (Guillaume Hoffmann)
• when cloning from http, use packs in a more predictable way (Guillaume Hoffmann)
• store global cache in bucketed format (Marcio Diaz)

Developer-related

• require and support GHC 7.4 to 7.10 (Ganesh Sittampalam)
• replace type witness CPP macros with plain Haskell (Eric Kow)
• hashed-storage is bundled into darcs (Ganesh Sittampalam)
• replace C SHA256 bindings with external libraries (Ganesh Sittampalam)
• move the bits of the datetime package we need into Darcs.Util.DateTime (Ganesh Sittampalam)
• build Darcs once rather than thrice. (Eric Kow)
• remove home page and manual from darcs' repository (Guillaume Hoffmann)
• run tests through cabal test (Ryan Desfosses)
• run fewer darcs-1 related tests in testsuite (Ganesh Sittampalam)
• Use custom replHook to fix cabal repl (Owen Stephens)
• darcs.cabal: make Haskell2010 the default-language for all stanzas (Ben Franksen)
• always compile with mmap support (Ganesh Sittampalam)
• new options subsystem (Ben Franksen)
• various cleanups, code restructurations and refactoring, haddocks (Will Langstroth, Owen Stephens, Florent Becker, Guillaume Hoffmann, Michael Hendricks, Eric Kow, Dan Frumin, Ganesh Sittampalam)

Issues resolved in Darcs 2.10

• issue346: implement "patience diff" from bzr (Jose Neder)
• issue642: Automatic detection of file renames (Jose Neder)
• issue822: generalized the IO Type for better error messages and exception handling (Ernesto Rodriguez)
• issue851: interactive mode for whatsnew (Dan Frumin)
• issue904: Fix record on Linux/FUSE/sshfs (fall back to sloppy locks automatically) (Nathaniel Filardo)
• issue1066: clone to ssh URL by locally cloning then copying by scp (Guillaume Hoffmann)
• issue1268: enable to write darcs init x (Radoslav Dorcik)
• issue1416: put log files in tempdir instead of in working dir (Ale Gadea)
• issue1514: send --minimize-context flag for send (Guillaume Hoffmann)
• issue1624: bucketed cache (Marcio Diaz)
• issue1828: file listing and working --dry-run for mark-conflicts (Guillaume Hoffmann)
• issue1987: Garbage collection for inventories and patches (Marcio Diaz)
• issue2181: put cache in $XDG_CACHE_HOME (~/.cache by default) (Guillaume Hoffmann)
• issue2193: make that finalizeTentativeChanges no longer run tests (Guillaume Hoffmann)
• issue2198: move repo testing code to Darcs.Repository.Test (Guillaume Hoffmann)
• issue2200: darcs replace complains if no filepaths given (Owen Stephens)
• issue2204: do not send mail by default (Eric Kow)
• issue2237: prevent patch index creation for non-hashed repos (Owen Stephens)
• issue2235: Accept RFC2822 dates (Dave Love)
• issue2246: add default boring entry for emacs session save files (Owen Stephens)
• issue2253: attempting to use the patch index shouldn't create it on OF repos (Owen Stephens)
• Issue2278: Document default value for --keep-date / --no-keep-date (Mark Stosberg)
• issue2199: getMatchingTag needs to commute for dirty tags (Ganesh Sittampalam)
• issue2247: move patch index creation into the job running code (Ganesh Sittampalam)
• issue2238: let optsModifier remove all occurrences of LookForAdds (Gian Piero Carrubba)
• issue2236: make 'n' an alias for 'q' in lastregret questions (Gian Piero Carrubba)
• issue2155: Expurgate the non-functional annotate --xml-output option (Dave Love)
• issue2248: always clean up rebase-in-progress state (Ganesh Sittampalam)
• issue2270: fixed darcs changes -i --only-to-files (Sebastian Fischer)
• issue2282: don't allow remote operations to copy the rebase patch (Ganesh Sittampalam)
• issue2287: obliterate -O doesn't overwrite existing file. (Radoslav Dorcik)
• issue2227: move the rebase patch to the end before an amend-record (Ganesh Sittampalam)
• issue2277: rebase suspend and unsuspend supports --summary. (Radoslav Dorcik)
• issue2311: posthook for 'get' should run in created repo (Sebastian Fischer)
• issue2312: posthooks for 'record' and 'amend-record' should receive DARCS_PATCHES (Sebastian Fischer)
• issue2163: new option for amend, select author for patch stealing. (Radoslav Dorcik)
• issue2321: when no patch name given, directly invoke text editor (Jose Neder)
• issue2320: save prompted author name in ~/.darcs/author instead of ./_darcs/prefs/author (Jose Neder)
• issue2250: tabbing in usageHelper - pad by max length of command name (BSRK Aditya)
• issue2309: annotate includes line numbers (Owen Stephens)
• issue2334: fix win32 build removing file permission functions (Guillaume Hoffmann)
• issue2343: darcs amend-record does not record my change (Jose Neder)
• issue2335: one liner when adding tracked files if not verbose (Guillaume Hoffmann)
• issue2313: whatsnew -l: Stack space overflow (Jose Neder)
• issue2347: fix amend-record --prompt-long-comment (Guillaume Hoffmann)
• issue2348: switch to cabal's test framework (Ryan Desfosses)
• issue2209: Automatically detect replace (Jose Neder)
• issue2332: ignore case of characters in prompt (Guillaume Hoffmann)
• issue2263: Option --set-scripts-executable is not properly documented (Ale Gadea)
• issue2367: rename amend-record to amend, make --unrecord more visible (Guillaume Hoffmann)
• issue2345: solution using cabal's checkForeignDeps (Dan Frumin)
• issue2357: switching to regex-compat-tdfa for unicode support (Dan Frumin)
• issue2379: only use packs to copy pristine when up-to-date (Guillaume Hoffmann)
• issue2365: correctly copy pristine in no-working-dir clones (Guillaume Hoffmann)
• issue2244: darcs tag should warn about duplicate tags (Ale Gadea)
• issue2364: don't break list of 'bad sources' (Sergei Trofimovich)
• issue2361: optimize --reorder runs forever with one repository (Ale Gadea)
• issue2364: fix file corruption on double fetch (Sergei Trofimovich)
• issue2394: make optimize a supercommand (Guillaume Hoffmann)
• issue2396: make convert a supercommand and enhance help strings (Guillaume Hoffmann)
• issue2314: output-auto-name in defaults file (Ben Franksen)
• issue2388: check if inventories dir has been created (Owen Stephens)
• issue2249: Rename isFile to isValidLocalPath and WorkRepoURL to WorkRepoPossibleURL (Mateusz Lenik)
• issue2153: allow skipping backwards through depended-upon patches (Andreas Brandt)
• issue2380: allow darcs mv into known, but deleted in working, file (Owen Stephens)
• issue2403: need to avoid moving the rebase patch to the end (Ganesh Sittampalam)
• issue2409: implement darcs rebase apply (Ganesh Sittampalam)
• issue2385: invoke pager without temporary file (Guillaume Hoffmann)
• issue2333: better error message when pushing and darcs not in path (Ben Franksen)

Known issues

These are known new issues in darcs 2.10.0:

• issue2269: rebase should warn about stolen patches at suspend, not unsuspend
• issue2272: darcs rebase unsuspend should automate or semi-automate handling unrecorded changes
• issue2276: darcs rebase unsuspend needs UI improvements for "You are not... Amend anyway?"
• issue2359: convert --export mishandles Unicode filenames
• issue2372: Please remove "HINT: I could not reach..." message
• issue2423: diff only respecting --diff-command when a diff.exe is present
• issue2436: rollback --patches takes ages before first prompt
• issue2445: internal error if suspended patch is pulled into repository again
• issue2449: test harness/shelly: need to handle mis-encoded/binary data

News and discussions

1. The Darcs 2.10 release is near! Please test the release branch and tell us if you find a bug.
   
   • http://darcs.net/Releases/2.10
   • http://lists.osuosl.org/pipermail/darcs-users/2015-March/027105.html
2. Joachim Breitner has shut down his darcswatch service after 7 years of activity:
   
   • http://lists.osuosl.org/pipermail/darcs-users/2015-February/027102.html
   • http://lists.osuosl.org/pipermail/darcs-users/2015-February/027104.html
   • http://www.joachim-breitner.de/blog/672-DarcsWatch_End-Of-Life%E2%80%99d

Issues resolved (11)

issue822 Ernesto Rodriguez

• Generalized the IO Type for better error messages and exception handling
• http://bugs.darcs.net/issue822

issue2260 Ganesh Sittampalam

• skip internal patches when counting 
• http://bugs.darcs.net/issue2260

issue2385 Guillaume Hoffmann

• invoke pager without temporary file 
• http://bugs.darcs.net/issue2385

issue2410 Guillaume Hoffmann

• detect replaces even with tokens of different lengths 
• http://bugs.darcs.net/issue2410

issue2411 Guillaume Hoffmann

• ignore swap moves with --look-for-moves 
• http://bugs.darcs.net/issue2411

issue2414 Guillaume Hoffmann

• use parseFlags instead of toMatchFlags where needed 
• http://bugs.darcs.net/issue2414

issue2418 Guillaume Hoffmann

• add and check on-disk patch index version 
• http://bugs.darcs.net/issue2418

issue2422 Ganesh Sittampalam

• drop unneeded LambdaCase extension 
• http://bugs.darcs.net/issue2422

issue2427 Ben Franksen

• start conversion from intermediate tree state 
• http://bugs.darcs.net/issue2427

issue2431 Ben Franksen

• rollback Control.Monad.Catch generalizations 
• http://bugs.darcs.net/issue2431

issue2432 Ganesh Sittampalam

• use the merged version of the local patches 
• http://bugs.darcs.net/issue2432

issue2437 Guillaume Hoffmann

• fix cloning to ssh by forgetting source repo later 
• http://bugs.darcs.net/issue2437

Patches applied (111)

See darcs wiki entry for details.

News and discussions

1. Darcs has received two grants from the Google Summer of Code program, as part of the umbrella organization Haskell.org. Alejandro Gadea will work on history reordering:
   
   • http://alegdarcs.blogspot.com.ar/2014/05/google-summer-of-code-2014-darcs.html
2. Marcio Diaz will work on the cache system:
   
   • http://marcioodiaz.blogspot.com.ar/2014/04/gsoc-project-accepted.html
   • http://marcioodiaz.blogspot.com.ar/2014/04/gsoc-progress-report-1-complete.html
3. Repository cloning to remote ssh hosts has been present for years as darcs put. This feature has now a more efficient implementation:
   
   • http://hub.darcs.net/darcs/darcs-reviewed/patch/20140425060647-5ef8f

Issues resolved (11)

issue851 Dan Frumin

• interactive mode for whatsnew
• http://bugs.darcs.net/issue851

issue1066 Guillaume Hoffmann

• clone to ssh URL by locally cloning then copying by scp
• http://bugs.darcs.net/issue1066

issue1268 Guillaume Hoffmann

• enable to write darcs init x
• http://bugs.darcs.net/issue1268

issue1416 Ale Gadea

• put log files in tempdir instead of in working dir.
• http://bugs.darcs.net/issue1416

issue1987 Marcio Diaz

• Garbage collection for inventories and patches.
• http://bugs.darcs.net/issue1987

issue2263 Ale Gadea

• Option --set-scripts-executable is not properly documented
• http://bugs.darcs.net/issue2263

issue2345 Dan Frumin

• solution using cabal's checkForeignDeps
• http://bugs.darcs.net/issue2345

issue2357 Dan Frumin

• switching to regex-compat-tdfa for unicode support
• http://bugs.darcs.net/issue2357

issue2365 Guillaume Hoffmann

• correctly copy pristine in no-working-dir clones
• http://bugs.darcs.net/issue2365

issue2367 Guillaume Hoffmann

• rename amend-record to amend, make --unrecord more visible
• http://bugs.darcs.net/issue2367

issue2379 Guillaume Hoffmann

• only use packs to copy pristine when up-to-date
• http://bugs.darcs.net/issue2379

Patches applied (41)

See darcs wiki entry for details.

News and discussions

1. We are in the feature freeze period of darcs 2.10:

• http://lists.osuosl.org/pipermail/darcs-users/2014-November/027056.html

2. Our two Summer of Code projects ended up two months ago. Marcio and Ale's code will be part of the upcoming new stable version of darcs. In case you missed them, here are the latest posts of Marcio for his project:

• http://marcioodiaz.blogspot.com.ar/2014/07/gsoc-progress-report-3-bucketed-global_23.html
• http://marcioodiaz.blogspot.com.ar/2014/07/gsoc-progress-report-4-garbage.html
• http://marcioodiaz.blogspot.com.ar/2014/07/gsoc-progress-report-5-starting.html

3. Ale's posts:

• http://alegdarcs.blogspot.com.ar/2014/07/month-of-june.html
• http://alegdarcs.blogspot.com.ar/2014/07/some-week-14-19-july.html
• http://alegdarcs.blogspot.com.ar/2014/07/other-week-21-26-july.html
• http://alegdarcs.blogspot.com.ar/2014/08/last-few-weeks.html

Issues resolved (7)

issue1514 Guillaume Hoffmann

• send --minimize-context flag for send
• http://bugs.darcs.net/issue1514

issue1624 Marcio Diaz

• bucketed cache.
• http://bugs.darcs.net/issue1624

issue2153 Andreas Brandt

• allow skipping backwards through depended-upon patches
• http://bugs.darcs.net/issue2153

issue2249 Mateusz Lenik

• Rename isFile to isValidLocalPath and WorkRepoURL to WorkRepoPossibleURL
• http://bugs.darcs.net/issue2249

issue2380 Owen Stephens

• allow darcs mv into known, but deleted in working, file
• http://bugs.darcs.net/issue2380

issue2403 Ganesh Sittampalam

• need to avoid moving the rebase patch to the end
• http://bugs.darcs.net/issue2403

issue2409 Ganesh Sittampalam

• implement darcs rebase apply
• http://bugs.darcs.net/issue2409

Patches applied (118)

See darcs wiki entry for details.

In the last three weeks I was working in a couple of things, unfortunately I couldn't complete any to 100% :

- Finishing the command minimize-context.
- Implement the "show dependencies" for darcsden.
- Solve the issue2405.

Finish the command minimize-context

I can start telling that the command itself is implemented, but making tests I find out a case when making minimize-context fails to update correctly the files of repository. I comment the problem in here, however here goes a little summary.

Some "preconditions" to take into account:
1. If exist a tag, I search for dependencies in the patchs after the tag.
2. If happends that not exist dependencies after the tag, the only patch in the context of the bundle to send is the tag. This helps the command darcs apply to merge the patches and seems good to have at least a "tag patch" dependency.
3. (In relation with 1) If not exist a tag, the search is made in all the repository. One problem here is that with repositories that have huge amount of patches (darcs for example with 11000~ patches) the command maybe not finish of calculate the dependencies. Hopefully, having so many patches without tags seems a little odd. Nevertheless, I suppose that even the search in 700~ patches of 10~ patches to send should have a decent performance.

Well, passing to comment the problem. In some older post I mention the way we calculate the dependencies, so with that in mind, suppose we have to repositories $r_1$, $r_2$ with the following patches, where $m_x$ and $a_x$ represents the $x$-th modification and adding respectively.

$r_1$,$r_2$ = $m_2$ $tag$ $m_1$ $a_1$

suppose now that $r_2$ makes a third modification $m_3$ that adds lines without touch any existing line in the only file adding for $a_1$ and sequentially modified for $m_x$. With the idea of send a bundle to $r_1$. Now if we compute the dependencies to find out what is the context to send. For (1) we need to try to commute $m_2$ with $m_3$, which success but of course the end result is a $m'_3$ and a $m'_2$ because the lines which modifies every one now are different. But beyond that for our definition of dependency we can throw $m_2$, so the final context of the bundle with minimize context is the patch $tag$.
The problem now is that if we make darcs apply of the minimized bundle the merge is made not considering $m_2$ and the final state of the file isn't right. For example, if we are talking about that $m_3$ adds a function in some empty place, this could end in $m_3$ adding the function in the middle of an existing function.

My, more or less, mature thought is something like have Direct Dependency and  Indirect Dependency (the names could be differents $\smile$). The Direct Dependency is a dependency as already we know it and the Indirect Dependency is that if we have,
$p_1$ and $p_2$ patches, and $p'_1$ and $p'_2$  are the results of commute the patches, then if $p_1 \ne p'_1$ we conclude that $p_1$ indirectly depends of $p_2$.

I have almost implemented this idea, but I change task and still missing a couple of details. In particular, I'm not so sure of a couple of things,
- How to correctly compare the patches ($p_1 \ne p'_1$)
- In my unclean implementation I have to carry on $MyEq$ for all over the code and it's very ugly I think.

Finishing with this topic, here is a script (improved by Guillaume) that shows the problem:

Link: http://hub.darcs.net/alegadea/ExamplesRepos
SH: getDepsDoesntAlwaysWork.sh

Implement the "show dependencies" for darcsden

An idea that quickly comes up to my mind is that, considering the last part of the previous topic, since we have two types of dependencies it would be nice generate the graph of dependencies with two types of arrows to differentiate the type of dependency between patches. Talking about the implementation itself, I advanced in the drawing of the graph and some different presentations, but very little about the integration with darcsden, which I think is more or less direct having implemented the $js$ that draw the graph. Would have been nice have more time for finish this but I swap to the task of the following topic.

I hope be able to work in the last two topics in the next few months outside of the GSoC.

Solve the issue2405

I would like to start saying, "what a shame...". I can't solve the issue, I improve a little bit the memory but nothing too much significant. Maybe the good news is that, more o less, I found the "place" in the code where the memory triggers "to the sky" :)
Now, the reality is that I tried so many things that I'm a little bit confused about everything. So, I gonna comment the last feelings and tests that I make and, any question(or idea) for the future person (maybe me) that would try to solve the issue, it would well received.
So things that I tried to change:

- Strict version of Map, Monad, etc.
- More strict version of mapM_, gets and modify.
- Some partials changes in the data type PatchIndex.
- Using darcs as library; differents uses of the functions...

The "place" that I suspect the code has the memory glitch is in $applyPatchMods \rightarrow go (pid, PTouch fn)$. I think that my lack of experience dealing with this kind of problem comes to light :) the
bright side is that I learn many things on the way of understand the problem and try different solutions.

Closing, maybe this could be my last post in a long time. I would like to thanks Guillaume and the darcs community in general. It was great to contribute to the development of darcs!

Again, I hope be able to work a little outside the GSoC, in particular in the "dependency problems and tasks".

Thanks!

Good news!

I almost finish to implement the option minimize-context for the command darcs send, I say almost because making some examples I find out that somethings could go wrong. But before entering in that, I will comment a little the main the test that without minimize-context doesn't work and with the option passes(when the implementation is ready the most likely is I will upload the example and some more in ExamplesRepos), and some "tentatives options" for darcs send that I think could be useful.

So, the minimal example:

Suppose we have repositories $r_1$, $r_2$ and $r_3$; $r_2$ a direct clone of $r_1$ where
originally $r_1$ has only one patch, 

$r_1$,$r_2$ $=$ $p_0$

now, we add a patch with a new file in $r_1$ and make a clone, $r_3$ which leave the repositories like this,

$r_1$,$r_3$ $=$ $p_0$ $p_1$

Suppose that we make a modification in the file that add $p_0$ and make a patch $p'_0$ in $r_1$ and we send a bundle (with darcs send) to $r_2$ and $r_3$. What ends up happening is what one imagine,
the bundle can be applied to $r_3$ but not to $r_2$, this because the bundle has de following "shape":

Some message

patch [Some hash]
Author: ...
Date: ...
  * [The modification to $p_0$]

New patches:

$p'_0$

Context:

$p_1$
$p_0$

Patch bundle hash: [Some hash]

Hi all!

I was finishing understanding and implementing the command darcs send --minimize-context using "optimize reorder" when I begin to suspect that doesn't solve the problem described in here. The thing is, despite the fact that the context in the bundle to send is reduced if before we send we make "optimize reorder", this doesn't solve the problem of dependencies. Guillaume finished of evacuate my doubts, and so after read:

- [darcs-users] darcs cannot apply some patch bundles
- irclog
- issue1514 (which is the issue which "replace" issue2044 darcs send should do optimize --reorder)

I convince myself of what needs to be done, and it's calculate the "exact" dependencies of the patches to send so such dependencies be the context in the bundle to send. "Exact" because for big repositories can be very costly and calculate till certain tag seem appropriate.

Now, one concern is the cost of doing the search of dependencies. About this I can first comment some of the things I was doing during the week and later show, what I think are, encouraging examples. So first, maybe the most relevant thing of the week it's the implementation of the command darcs show dependencies with the following "description":

Usage: darcs show dependencies [OPTION]... 
Generate the graph of dependencies.


Options:
              --from-match=PATTERN  select changes starting with a patch matching PATTERN
              --from-patch=REGEXP   select changes starting with a patch matching REGEXP
              --from-tag=REGEXP     select changes starting with a tag matching REGEXP
              --last=NUMBER         select the last NUMBER patches
              --matches=PATTERN     select patches matching PATTERN
  -p REGEXP   --patches=REGEXP      select patches matching REGEXP
  -t REGEXP   --tags=REGEXP         select tags matching REGEXP
              --disable             disable this command
  -h          --help                shows brief description of command and its arguments

till the moment the command returns a graph described in dot language, this can eventually change. But with the current returned value one can do:

$\$$ darcs show dep | dot -Tpdf -o deps.pdf

to draw the graph in a pdf. Finally, in summary to calculate the dependencies I use more or less the idea which describes Ganesh in here.

Moving to the examples is interesting, thinking in the performance of the implementation of darcs send --minimize-context using this approach, to see the followings results:

1. Show the dependencies after the tag 2.9.9 (75 patches)
$ time darcs show dep --from-tag=2.9.9
real 0m0.397s
user 0m0.373s
sys 0m0.026s
darcsDesps299.pdf
real 0m2.951s
user 0m2.865s
sys 0m0.082s
darcsDesps298.pdf

3. Show the dependencies after the tag 2.9.7 (288 patches)
$ time darcs show dep --from-tag=2.9.7
real 0m26.654s
user 0m26.003s
sys 0m0.511s
darcsDesps297.pdf

4. Show the dependencies after the tag 2.9.6 (358 patches)
$ time darcs show dep --from-tag=2.9.6
real 0m39.019s
user 0m38.302s
sys 0m0.666s
darcsDesps296.pdf
5. Show the dependencies after the tag 2.9.5 (533 patches)
$ time darcs show dep --from-tag=2.9.5
real 1m53.730s
user 1m51.343s
sys 0m1.939s
darcsDesps295.pdf

News and discussions

1. Google Summer of Code 2013 has begun! BSRK and José will post updates on their blogs:
   
   • http://bsrkaditya.blogspot.com/search/label/darcs
   • http://blog.jlneder.com.ar/search/label/darcs

Issues resolved (8)

issue2163 Radoslav Dorcik

• new option for amend, select author for patch stealing.
• http://bugs.darcs.net/issue2163

issue2227 Ganesh Sittampalam

• move the rebase patch to the end before an amend-record
• http://bugs.darcs.net/issue2227

issue2248 Ganesh Sittampalam

• always clean up rebase-in-progress state
• http://bugs.darcs.net/issue2248

issue2250 BSRK Aditya

• tabbing in usageHelper - pad by max length of command name
• http://bugs.darcs.net/issue2250

issue2311 Sebastian Fischer

• posthook for 'get' should run in created repo
• http://bugs.darcs.net/issue2311

issue2312 Sebastian Fischer

• posthooks for 'record' and 'amend-record' should receive DARCS_PATCHES
• http://bugs.darcs.net/issue2312

issue2320 Jose Luis Neder

• save prompted author name in ~/.darcs/author instead of ./_darcs/prefs/author
• http://bugs.darcs.net/issue2320

issue2321 Jose Luis Neder

• when no patch name given, directly invoke text editor
• http://bugs.darcs.net/issue2321

Patches applied (20)

See darcs wiki entry for details.

News and discussions

1. This year's Google Summer of Code projects brought a lot of improvements to darcs and its ecosystem!
   
   • BSRK Aditya: Darcsden improvements:
     
     • http://joyful.com/blog/2013-09-26-darcsden-darcs-hub-gsoc-complete.html
     • http://bsrkaditya.blogspot.com.ar/2013/09/darcsden-gsoc-final-blog-post.html
   • José Neder: patience diff, file move detection, token replace detection:
     
     • http://lists.osuosl.org/pipermail/darcs-users/2013-October/026945.html
     • http://blog.jlneder.com.ar/search/label/darcs
2. Gian Piero Carrubba asked why adjacent hunks could not commute:
   
   • http://lists.osuosl.org/pipermail/darcs-users/2013-November/026958.html
3. We listed the changes that occurred between version 2.8.4 and the current development branch into a 2.10 release page:
   
   • http://darcs.net/Releases/2.10?revision=20131028184855-5ef8f-ba33870773ad9ee010a470ac8536b5c7ae331b50

Issues resolved (8)

issue346 Jose Luis Neder

• implement "patience diff" from bzr
• http://bugs.darcs.net/issue346

issue1828 Guillaume Hoffmann

• file listing and working --dry-run for mark-conflicts
• http://bugs.darcs.net/issue1828

issue2181 Guillaume Hoffmann

• put cache in $XDG_CACHE_HOME (~/.cache by default)
• http://bugs.darcs.net/issue2181

issue2309 Owen Stephens

• annotate includes line numbers
• http://bugs.darcs.net/issue2309

issue2313 Jose Luis Neder

• whatsnew -l: Stack space overflow
• http://bugs.darcs.net/issue2313

issue2334 Guillaume Hoffmann

• fix win32 build removing file permission functions
• http://bugs.darcs.net/issue2334

issue2343 Jose Luis Neder

• darcs amend-record does not record my change
• http://bugs.darcs.net/issue2343

issue2347 Guillaume Hoffmann

• fix amend-record --prompt-long-comment
• http://bugs.darcs.net/issue2347

Patches applied (39)

See darcs wiki entry for details.

News and discussions

1. Darcs is participating once again to the Google Summer of Code, through the umbrella organization Haskell.org. Deadline for student application is Friday 21st:
   
   • http://darcs.net/GSoC
   • https://www.google-melange.com/gsoc/document/show/gsoc_program/google/gsoc2014/help_page
2. It is now possible to donate stock to darcs through the Software Freedom Conservancy organization. Donations by Paypal, Flattr, checks and wire transfer are still possible:
   
   • http://lists.osuosl.org/pipermail/darcs-users/2013-December/026977.html
   • http://darcs.net/Donations
3. Dan Licata wrote a presentation about Darcs as a higher inductive type:
   
   • http://lists.osuosl.org/pipermail/darcs-users/2014-January/026985.html
4. Darcs now directly provides import and export commands with Git. This code was adapted from Petr Rockai's darcs-fastconvert, with some changes by Owen Stephen from his Summer of Code project "darcs-bridge":
   
   • http://hub.darcs.net/darcs/darcs-screened/patch/20140125154537-5ef8f

Issues resolved (6)

issue642 Jose Luis Neder

• Automatic detection of file renames
• http://bugs.darcs.net/issue642

issue2209 Jose Luis Neder

• Automatically detect replace
• http://bugs.darcs.net/issue2209

issue2319 Guillaume Hoffmann

• make changes non-interactive if fed to a pipe
• http://bugs.darcs.net/issue2319

issue2332 Guillaume Hoffmann

• ignore case of characters in prompt
• http://bugs.darcs.net/issue2332

issue2335 Guillaume Hoffmann

• one liner when adding tracked files if not verbose
• http://bugs.darcs.net/issue2335

issue2348 Ryan

• switch to cabal's test framework
• http://bugs.darcs.net/issue2348

Patches applied (34)

See darcs wiki entry for details.

News and discussions

1. We have a few updates from the Google Summer of Code projects. Alejandro Gadea about history reordering:
   
   • http://alegdarcs.blogspot.com.ar/2014/05/first-week-19-23-may.html
   • http://alegdarcs.blogspot.com.ar/2014/06/second-week-26-30-may.html
   • http://alegdarcs.blogspot.com.ar/2014/06/third-week-02-06-june.html
2. Marcio Diaz about the cache system:
   
   • http://marcioodiaz.blogspot.com.ar/2014/06/gsoc-progress-report-2-bucketed-global.html
3. Incremental fast-export is now provided to ease maintenance of git mirrors:
   
   • http://hub.darcs.net/darcs/darcs-screened/patch/20140609190214-5ef8f

Issues resolved (8)

issue2244 Ale Gadea

• darcs tag should warn about duplicate tags
• http://bugs.darcs.net/issue2244

issue2314 Benjamin Franksen

• output-auto-name in defaults file
• http://bugs.darcs.net/issue2314

issue2361 Ale Gadea

• optimize --reorder runs forever with one repository
• http://bugs.darcs.net/issue2361

issue2364 Sergei Trofimovich

• fix file corruption on double fetch
• http://bugs.darcs.net/issue2364

issue2364 Sergei Trofimovich

• don't break list of 'bad sources'
• http://bugs.darcs.net/issue2364

issue2388 Owen Stephens

• check if inventories dir has been created
• http://bugs.darcs.net/issue2388

issue2394 Guillaume Hoffmann

• make optimize a supercommand
• http://bugs.darcs.net/issue2394

issue2396 Guillaume Hoffmann

• make convert a supercommand and enhance help strings
• http://bugs.darcs.net/issue2396

Patches applied (39)

See darcs wiki entry for details.

Well, well... Now with the solution already implemented here are a couple of time tests that show the improvement.

For the repository of the issue2361:

For the repository generated by forever.sh, that in summarize has 12600~ patches, a bundle unrevert and doing reorden implies move 1100~ patches forward passing by 11500~ patches.


The repository generated by bigRepo.sh has 600~ patches, with only one tag and a very small bundle unrevert.

One last repository generated by bigUnrevert.sh, has 13 patches and a really big bundle unrevert (~10MB).

Luckily, this week with Guillaume we found a "solution" for the issue 2361. But before of entering in details, let's review how the command darcs optimize --reorder does for reorder the patches.

So, suppose we have the following repositories than, reading it from left to right we have the first patch till the last patch, besides with $p_{i,j}$ we denote the $i$-th patch who belongs to the $j$-th repository, and when we want to specify that a patch $p_{i,j}$ is a tag we write $t_{i,j}$.

$r_1$ $=$ $p_{1,1}$ $p_{2,1}$ $\ldots$ $p_{n,1}$ $p_{n+1,1}$ $\ldots$ $p_{m,1}$

$r_2$ $=$ $p_{1,1}$ $p_{2,1}$ $\ldots$ $p_{n,1}$ $p_{1,2}$ $\ldots$ $p_{k,2}$ $t_{1,2}$ $p_{k+1,2}$ $\ldots$ $p_{l,2}$

where the red part represent when $r_2$ was cloned from $r_1$, and the rest is how each repository was evolved. Now, suppose we make a merge of $r_1$ and $r_2$ in $r_1$ making a bundle of the patches of $r_2$ and appling it in $r_1$. Thus, after the merge we have that

$r_1$ $=$ $p_{1,1}$ $p_{2,1}$ $\ldots$ $p_{n,1}$ $p_{n+1,1}$ $\ldots$ $p_{m,1}$ $p_{1,2}$ $\ldots$ $p_{k,2}$ $t_{1,2}$ $p_{k+1,2}$ $\ldots$ $p_{l,2}$

and we found the situation where the tag $t_{1,2}$ is dirty because the green part is in the middle. And now we are in conditions of finding out how darcs does the reorder of patches.
So, the first task is to select the first tag seeing $r_1$ in the reverse way, suppose $t_{1,2}$ is the first (ie, $p_{k+1,2}$ $\ldots$ $p_{l,2}$ are not tags), and split the set of patches (the repository) in

$ps_{t_{1,2}}$ $=$ $p_{1,1}$ $p_{2,1}$ $\ldots$ $p_{n,1}$ $p_{1,2}$ $\ldots$ $p_{k,2}$ $t_{1,2}$

and the rest of the patch set,

$rest$ $=$ $p_{n+1,1}$ $\ldots$ $p_{m,1}$ $p_{k+1,2}$ $\ldots$ $p_{l,2}$

this is done by splitOnTag, which I don't totally understand yet, so for the moment... simply do the above :) Then, the part that interest us now is $rest$, we want to delete all the patches of $rest$ that exist in $r_1$ and then add them again, causing that they show up to the right. This job is done by tentativelyReplacePatches, which first calls tentativelyRemovePatches and then calls tentativelyAddPatches.

So, tentativelyRemovePatches of $r_1$ and $rest$ makes,

$r_{1}'$ $=$ $p_{1,1}$ $p_{2,1}$ $\ldots$ $p_{n,1}$ $p_{1,2}$ $\ldots$ $p_{k,2}$ $t_{1,2}$

and, tentativelyAddPatches of $r_{1}'$ and $rest$,

$r_{1}''$ $=$ $p_{1,1}$ $p_{2,1}$ $\ldots$ $p_{n,1}$ $p_{1,2}$ $\ldots$ $p_{k,2}$ $t_{1,2}$ $p_{n+1,1}$ $\ldots$ $p_{m,1}$  $p_{k+1,2}$ $\ldots$ $p_{l,2}$

leaving $t_{1,2}$ clean.

Well, all of this was for understanding the "solution" for the issue, we are almost there but before let's look at the function tentativelyRemovePatches. It attempts to remove patches with one special care: when one does darcs revert, a special file is generated, called unrevert in _darcs/patches, which is used for darcs unrevert in case that one makes a mistake with darcs revert. One important difference with unrevert is that unlike all the other files in _darcs/patches, unrevert in not a patch but a bundle, that contains a patch and a context. This context allows to know if the patch is applicable. So when one removes a patch (running for example oblitarete, unrecord or amend) that patch has to be removed from the bundle-revert (bundle of the file _darcs/patches/unrevert). It's now always possible to adjust the unrevert bundle, in which case, the operation continues only if the user agrees to delete the unrevert bundle.

But now a question emerge. Is it necessary to accommodate the bundle-revert in the case of reorder?; the answer is no, and it's because we don't delete any patch of $r_1$ so we still can apply the bundle-revert in $r_{1}''$.

So, finally! we find out that for reorder we need a special case of removing, which doesn't try to update the unrevert bundle. And this ends up being the "solution" for the issue, since the reorder blocks in that function. But! beyond this solves the issue something weird is happening, that is the reason of the double quotes for solution :)

This is more o less the step forward for now. The tasks ahead are, documenting the code in various parts and make the special case for the function tentativelyRemovePatches. On the way I will probably understand more about some of the functions that I mention before so probably I will add more info and rectify whatever is needed.