Monday, November 05, 2007

Pydev 1.3.10 released

This release did take a bit more than usual, mostly because the JDT integration proved more difficult than I had expected, and also, a bit too much was planned for a single release (I guess that at least a lesson was learned here)...

The JDT integration isn't totally finished: it's missing referencing a Java project from a Pydev project. There are some things that I still don't know how to treat correctly: Forum message (registration required)...

On the good side, some long awaited feature were added to this release:
- eggs/zips and symlinks are supported!
- JDT is used for code-completions / find definition (Extensions) for jars.
- Code-folding is available for elements such as while, if, etc.


Sunday, October 21, 2007

JDT Integration, zips and symlinks in Pydev

Right now I'm working on the pydev/JDT integration, so that pydev is able to provide a better code-completion for those working with jython.

I've just checked-in some code that is able to use JDT to gather completions for the jars defined in the jython interpreter. This fixes some things in the current integration, but it still can't do code-completion for project references (so, if you're working on a jython project that only depends on jars, that wouldn't be a problem, but if that's needed, there's still some work to do in that area).

Other areas worked recently (which should be available on a new release):
- Zip files are now supported in pydev.
- Symlinks are supported in the system pythonpath (but not in the project itself).

This release is taking a bit longer than usual, mostly because those changes are in some core areas, which makes them a bit more difficult, as lots of code depends on it -- good thing all the tests are still running ;-)

Also, a good approach to integrate JDT when it comes to java projects referenced from jython projects is something that's proving a bit more difficult than initially thought -- so, the next release will probably not have it... still, it's planned to be available soon after that.

Saturday, September 01, 2007

Debugging Eclipse

Right now it appears that Eclipse sporadically does not shut down correctly for me: some times it is loosing my view configurations (they are not saved as they should be) and a process keeps running in the background.

Also, I can't seem to reproduce it reliably: it just happens in my production environment after running Eclipse for quite some time... usually after 1 or 2 days open.

So, I've been looking for a way to debug the Eclipse shutdown to see if there's something odd there, and the best way I was able to find is starting Eclipse with some debug flags:

-Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n

And later, I can attach a debugger to that running instance (in my case, I'll start another Eclipse process and debug it using the 'remote java application' configuration in the run dialog).

The downside is that it is a bit slower to run Eclipse (but just a bit) and I'm not sure if using those options will make the vm unstable or not, but at least I'll be able to debug things in a production environment!

Or maybe there's a better way to do it?

Some useful references I found about it:

Eclipse remote debugging
Getting runtime info (as a snapshot) on a vm

Tuesday, August 14, 2007

Planning for the next release

Ok, the next release is taking some more time than usual, but that's because I'll be out next week, and I don't want to make a release before I come back -- making a release and not being able to support it the week after is usually not good -- and also, the current release is pretty stable...

Ok, there are some bugs, but the most annoying already have descriptions on how to fix them with the current release:
- Jython debugger halting: link to sourceforge bug
- Not being able to configure interpreter: link to sourceforge bug

So, the next release is targeted to the end of this month. It will focus a lot on those little details -- that can usually make programming much more enjoyable -- and some bug-fixes (as usual).

Wednesday, July 18, 2007

Pydev 1.3.7 and 1.3.8

Well, I didn't blog about 1.3.7 before -- and now 1.3.8 is already out ;-)

Both releases are pretty much focused on bug-fixing, solving some issues regarding the integration with Eclipse 3.3 and java 1.6.

Other noteworthy things are:

The code-completion engine uses some more heuristics for type inference (together with some other approaches -- which are pretty much the same thing). So, the following is now considered:
- Python: assert isinstance(xxx, Class)
- PyProtocols: adapt(obj, Class)
- Zope: assert Interface.implementedBy(obj)

And the Pydev Extensions Code-Analysis now detects mixing of tabs/spaces.

Thursday, July 05, 2007

Problems in pydev 1.3.6

Ok, there have been some reports of errors in 1.3.6

It's still the same error that should be actually fixed in 1.3.6 -- the builtins completions are not always there (and another problem that seems related to mylyn).

This should be fixed in the next release (which should be out in the beggining of the next week -- together with the integration for mylyn and support for eclipse 3.3).

So, if possible, it's reccomended to keep with Eclipse 3.2.x and pydev 1.3.4 until that release comes out.

Friday, June 29, 2007

Pydev 1.3.6 Released

This was mainly a bug-fix release.

The major problem was that in new configurations it would not set the forced builtins correctly internally. The effect was that no builtins would appear in code-completion or when doing code-analysis.

Aside from that, a patch provided by Carl Robinson allows users to config PyLint severities.

Wednesday, June 27, 2007

Pydev and JDT (SDK not required anymore)

I forgot to mention about it... Pydev 1.3.5 does not require JDT to work anymore (it's set as an optional dependency, only needed for jython development), so, users that are interested only in python can download the Eclipse Platform Runtime Binary instead of the whole SDK.

There's only one minor problem with that: the platform does not include the Error Log (which is usually useful when something goes wrong for bug reports) -- so, if you're using only the runtime and not the whole SDK, you need to look through the .metadata/.log file for those errors.

Another option is getting only the plugin that adds the error log view: org.eclipse.pde.runtime_XXX.jar -- it's pretty small (147kb here), but it needs to be gotten in the SDK.

Friday, June 22, 2007

Noteworthy things about Pydev 1.3.5

Ok, the new version was released yesterday, and there are a couple of things worth mentioning:


  • Docstrings and the pop-up window: the docs are now correctly wrapped, those whitespace columns to the left are removed and the size is kept among code-completion requests.

  • Outline: the comments handling is much better (it respects the level they should appear in, finds comments ending with '---' and sorts them by their position, even when in alphabetic order is chosen)

Another thing that had changes was the debugger (as always)... it should be working with jython (some semantics are different within python and jython regarding the tracing facility, so, it was already broken for about 3 releases -- hopefully things should be working again).

wxPython debugging also works again... it was not working correctly in the last release because pydev was keeping the latest frame from a thread alive a bit more than it should (and wxPython didn't like it). That was needed because pydev currently runs with several untraced frames, but to make them traceable again, pydev needs some way to access them (so, it kept the topmost frame available in the thread)... that access is still needed, so, 2 workarounds for that are now in place to avoid that strong reference to the topmost frame:

  • python 2.5 added support to get the running frames for all the threads in sys._current_frames(), so, this was a piece of cake (after I discovered about it).

  • Earlier versions need a different approach: a list of weak-refs to the PyDBFrame is kept (that's the class that wraps the tracing facility for a given frame) and that class has the actual frames as a strong ref (because you can't create actual weak-refs to frames)... seems like a hack for me, but the only other option required to have a compiled library to get the frames as python 2.5 does, but I didn't want to add the burden of having compiled code for different platforms (and that also didn't consider jython)

Saturday, June 16, 2007

Why can't the pydev debugger work with turbogears?

Ok, there's a problem that can't really be overcome in the pydev debugger when using turbogears... (just doing "import turbogears" would already break it).

Actually, no OPTIMIZED debugger would be able to work with that. I'm saying optimized because the implementation seems to take into account naive debuggers which would trace all the calls within all the frames (pydev only traces frames with breakpoints).

The problem is: there's a module that turbogears uses (in my tests: DecoratorTools-1.4-py2.5.egg) which has a decorator named: decorate_assignment. This decorator uses the tracing facility that python provides for debuggers and removes the current debugger tracer function. It still tries to restore it if it was tracing the frame previously (but that would hardly ever happen in an optimized debugger).

So, there's no way to actually fix that from pydev, but there are some options to make it work:

1. Using the pydev extensions remote debugger (but if that decorator is called after the remote debugger is set, the debugger would stop working again, so, this option would only useful if that decorator is not used later).

2. Removing that decorator from the places that use it in turbogears (the implications for that would have to be checked).

3. Hard-coding it to return the pydev tracing function. To do that, the file: DecoratorTools-1.4-py2.5.egg\peak\util\ must be changed so that the function "def decorate_assignment(callback, depth=2, frame=None):" does not use the call:

"oldtrace = [frame.f_trace]"

and uses the code below instead:

oldtrace = None
    import pydevd
    if debugger is not None:
        oldtrace = [debugger.trace_dispatch]

except ImportError:

if oldtrace is None:
    oldtrace = [frame.f_trace]

The 3rd option is probably the easier in the short run for those wanting to debug turbogears in pydev, but I think that the 2nd should be the one actually used (as a general rule, I believe that only debuggers should play with the tracing facility, because it tends to bee way to instrusive, and it's probably the most un-optimized way of doing something, as you're going to trace all that happens, which can lead to a large overhead).

Thursday, June 14, 2007

Working offline in the pydev source

Ok, after quite some time being really annoyed at not being able to commit when I want to the cvs at sourceforge, and sync operations taking almost forever sometimes (yeap, I double-check everything before commiting), I've decided to take a look at alternative options.

Basically, descentralized scm systems seem the way to go, so, after taking a (rather quick) look at some of the alternatives (which were: mercurial, git and bzr), mercurial apparently seems what'll go better with how I want to work.

It seems to be able to coexist nicely with the base in place -- that's a must because I'll have to keep commiting things to the cvs at sourceforge. And it's pretty non-obtrusive (it doesn't keep zilions of files around each folder as svn and cvs do. All is kept under a single folder, out of the way of the actual projects -- and having a single .hgignore file instead of one for each folder is also pretty nice)

All my projects are in a X: drive (when in windows), so, basically, I've simply created a mercurial repository at X: and imported all the pydev stuff into it.

I've just played with it for about 1-2 hours, and I already feel I cannot live without it anymore ;-)

I'll probably change my modus-operandi to make all things in my machine, with patches and diffs (BTW: I'm using KDiff3 which integrates nicely with Mercurial for that) and just commiting everything to sourceforge at once, without having to double check all those things...

But the most important thing is: I feel really relieved of not having to make those syncs that took forever to see if everything is correct when commiting to sourceforge (and working offline is a big win too).

The major drawback is that the integration with Eclipse is still in its early stages (in fact, I'll probably be using the command line and diffs with KDiff3 until it matures -- which I hope will not take long) -- but that's still a minor thing when compared with the advantages.

Thursday, May 31, 2007

Pydev release 1.3.4 and first impressions on Eclipse 3.3

Ok, 1.3.4 is out. The major changes were on the debugger (fixing bugs that were introduced in 1.3.3, given some major refactorings in its structure -- mostly to achieve a better performance -- so, all should be working again).

Also, I've downloaded and installed Eclipse 3.3 to start trying pydev with it. Mostly the experience has been nice...

The new diff is what I liked best. It now gives you not only the lines that changed on a first glance, but also shows you what actually changed, without having to go to 'next change' all the time -- pretty nice.

There were also some new things to learn: the startup.jar is now gone, so, if you do headless builds like me, you have to use eclipsec.exe instead of java -jar startup.jar -- which is something nice, but knowing which jar it uses is still something that may be handy (if you're doing profilings and need that jar, you can probably go to the configuration details and see how it was actually launched to use the same command-line -- haven't tested it, but it should work).

And the weirdest thing I found was that Ctrl+F11 doesn't always rerun the last run, it tries to find a configuration or something like that (probably for the project?) -- as I do tons of unit-tests and usually just do F9 in pydev to run the current editor and then just do Ctrl+F11 to re-run it again, I think that the modus operandi was nicer before for me -- fortunately, that is something that can be set in the preferences, so, I got to the old style and it works again as it used to.

Also, it seems that the java process is still alive when I finish eclipse. That may actually be a pydev bug (I've already had a report about it), so, I'll start to check if I can fix it (or if that's unrelated to pydev).

All in all, pydev seems to be behaving nicely with Eclipse 3.3 (aside from that bug), I've been able to build it without any code-changes, so, it seems pretty backward-compatible (but that's something that only time will tell).

Saturday, May 26, 2007

Latest Development News (and pydev release 1.3.3)

Ok, I didn't blog about 1.3.3, and now it's almost time for 1.3.4 ;-)

Anyway, I've already fixed a number of things from that release... I'm always amazed how those tiny little details can get you bugs you wouldn't think of...

This time a test-case for the debugger was needed to fix a bug. Making a test usually pays of, not only in the long run -- which is pretty clear to many -- but also in the short run, because having to start your application and manually having to reproduce some strange behaviour is too painful (and yes, the fix was a one-line fix, but usually, the time to fix things after finding the error is short -- the difficult thing is finding it, and preventing from repeating it later on).

Testing the debugger needed a bit of work... Basically the test was created through recording what was going on the socket from eclipse and then reproduced in a test as if it was a regular run... The debugger is one of those things people think it's very hard to test (actually, until you're used to testing, everything seems hard), but the result was pretty neat in the end (although it needs some refactoring before extending it for other tests).

There were some other fixes regarding the debugger too, so, if you use it, I really recommend upgrading to 1.3.4 ...(as soon as it's out)...

Also, the debugger it's now officially ultra-fast -- really, it's about a gazzilion times faster than the initial versions and almost the double from a month ago... until 1.3.2 it traced all the context in a file that had breakpoints, and 1.3.3 takes this a step further and only traces function contexts that have breakpoints -- aside from some other improvements.

Thursday, May 03, 2007

Debugging Zope in Pydev Extensions

I had already posted a message about the files that you need to add to the zope Products folder (see previous post), and I've just gotten feedback from Mario Orlandi with some additional things that need to be configured for debugging zope:

1) Add to \ the following statements:

# allows pydevd.set_trace() inside Script (Python) objects
from AccessControl import ModuleSecurityInfo

2) Install this Zope product:

3) Take a look at the following post:, which may also work as a good reference (and don't forget to start the zope and the pydev daemon from eclipse).

Saturday, April 21, 2007

New release for the mylar integration (0.2.0)

The new version of mylar (2.0.0.v20070403-1300) had some changes in its structure so that it can work with any element that adapts to IResource (which the pydev elements did), so, the mylar items in the menu were appearing twice (and this release fixes that).

Currently, it's only in the update site, but I intend to add it to the sourceforge downloads shortly.

Tuesday, April 17, 2007

Pydev 1.3.2 is out

A couple of features were adedd:

- Backspace is now indentation aware (I know some people have been waiting for that for some time).

- When having multiple editors opened with the same file, they will all share the same underlying parser (which is a nice optimization for this case).

Also, a number of bugs have been fixed... no worries here... there's still no risk of running out of bugs -- but at least, they keep on getting more elaborate ;-)

Current stats say that there are 16 still open (out of 478 already reported), but from those, some are open just as reminders (because they are out of the pydev scope, and depend on a solution at eclipse, java, python, etc). I'm not sure if it's better the way it is or if a known bugs should be somewhere... still, they should be around somewhere where users can find them easily (maybe the faq is a nice place for those).

Tuesday, March 27, 2007

Pydev 1.3.1 released (and Mylar integration)

Bug-fixes for everyone ;-)

I'm finally getting into how the package explorer should be used (after lots and lots of pitfalls)... the flat view is still not there, but all the other things should be already working -- I'm using it everyday, and have not seen a glitch in it in quite some, time, so, I think it can already be considered stable.

There's also a feature created for the mylar integration (you still have to add *.pyc files to the resources that mylar should ignore -- I've already submitted them a patch so that this is done automatically when using it), but aside from that it should be working well (it was actually pretty easy to integrate -- but it's still only integrated in the module level and not in the class/method level -- which I actually don't find myself using very much when using java anyways, so, I don't consider that a big hindrance).

So, enjoy...

Wednesday, March 21, 2007

Strange Out-of-memory errors

In the last couple of days (ever since I installed mylar -- which is a great plugin for managing tasks and related contexts -- but currently only integrates with the navigator in pydev), eclipse has been hard-crashing... and the error log said: java.lang.OutOfMemoryError: PermGen space (this is in jdk 1.5... in jdk 1.4 it would not say 'PermGen space', and would just say 'OutOfMemoryError'), so, after searching for it, it appears that there is some memory that the jvm uses just to store class definitions and other things that are permanent in the application, so, raising the usual -Xmx256m flag does not cover for it... you need to set: "-XX:MaxPermSize=128m" to raise that memory (the default is 64 mb).

This is probably only an issue for people that use many plugins (like I do), and therefore have many classes loaded in the jvm... so, it's nice to keep an eye open about it ;-)

Monday, March 12, 2007

Fresh pydev release: 1.3.0

Yeap, it's out... the package explorer now only has 1 minor glitch (still related to the actions, because they are not correctly synched with the selection in the tree, but it won't apply it wrong or anything -- just sometimes it will expand an item in the tree when it shouldn't -- hopefully that's the last one, as the others should be already fixed in this release)

And the extensions had one nice feature added: Find Occurrences: just select some word in the editor and press Ctrl+Shift+G and wait up to see the occurrences in your workspace (if you do that and then make a replace in the search view, it is almost the same as the rename refactoring -- but some people might like that better because you can cycle with Ctrl+. and review the occurrences in an open editor -- which sometimes in nicer than in the refactoring wizard)

The only drawback in that approach is if you want to rename a module... the search view still has to be customized to show python specific info, such as the module, class or method where the occurrence was found, etc -- because currently it will treat as if it was the return of a text-search and it would be nice if that context info could be shown in the search result -- but as it is, it's already quite handy.

Sunday, March 04, 2007

Pydev 1.2.9 released

Yeap... released just a few hours after 1.2.8 (to fix a packaging error).

Thursday, March 01, 2007

Packaging error in 1.2.8

Ok, there's a packaging error in 1.2.8 -- basically, the retroweaver jar was not included correctly in the org.python.pydev.refactoring plugin... It should be fixed shortly...



Pydev 1.2.8 released

Ok, the new release is out.

It has a number of nice things, with some particularly interesting highlights -- just a few will be commented on, so, check the pydev and the pydev extensions homepage for more details.

- Code-completion: many minor and major details were taken care of... from the fixes to make it work with the emacs mode from the 'always on' mode -- which starts the completion whenever you start typing...

- PEPTIC refactoring engine integration... it substitutes the extract method from BRM, and adds some other nice refactorings (Alt+Shift+T will bring a pop-up with the available ones).

- Package Explorer: keeps getting better ;-)


Tuesday, February 13, 2007

Pydev release 1.2.7

It's out -- or at least partially...

The homepage and update site at sourceforge are not updated because the sourceforge shell is currently offline (, so, as soon as it's back the release should be completed...

Still, it is already available in the update site at and as zip files at sourceforge.

This release enables changing a value in the debugger, and the pydev package explorer can now be safely used, as default actions (copy, paste, rename, etc) were reimplemented (see previous comment to see what this fixed) -- so, the package explorer is now almost out of the beta state.

Some high-priority fixes also made it into this release: the PYTHONPATH passed to PyLint now contains the project PYTHONPATH and a halting condition on code-completion is gone.

As for the pydev extensions, the major change is that the rename refactoring now works in comments and strings (and occurrences are also marked on those).

Friday, February 09, 2007

Notes on configuring Zope and Pylons

Just wanted to note that recently I added a (rather short) how-to on configuring Pylons in the sourceforge forum

And for those configuring Zope, there is one little catch... Zope actually has multiple locations with a 'Products' folder (and the parent of those Products folder must be added to the pythonpath... to the system pythonpath if it is a part of zope or to the project pythonpath if it is a part of your project)...

Now, the catch is that the Products folder does not have an file... so, pydev won't recognize it as a valid python package unless an file is added to each of the Products folder...

Apparently zope does some 'hacking' with the pythonpath itself and adds some black-magic to the mix instead of using the regular python import structure, to have multiple 'Products' scattered around (python would usually find the first one and not get the other Products)... still, it seems that adding those files won't have any side-effect on zope itself (and will enable code-completion / code analysis for your projects).

Update: some more details were added to:

Thursday, January 25, 2007

Pydev release 1.2.6

Ok, the release is out, and it seems everything is well ;-)

One source of complaints was that some had updated pydev and not pydev extensions, and this was not working well... so, I've taken a look and the problem is that when I specified the pydev extensions plugin, I asked for a 'greateOrEqual' match on the version, when a 'perfect' match is what's needed (otherwise, you could get a bunch of error with accesses to invalid methods, attributes, etc)

So, this is fixed for the next release... but until there, those that use both have to remind to update both.

Now, planning for the other release...

One of the nicest things added in the extensions is probably the rename refactoring (which is out, but still in beta), so, I'll probably try to improve the rename refactoring to include matches on strings and comments, and an option to match 'imprecise matches', so that the user can review other potential matches that the pydev analysis ended up discarding.

On the pydev side, I really wanted to go more for the package explorer... it has one issue that I think is critical (commented on: a previous post), and the way to go is probably recreating all the resource-based actions myself... as this can be tricky, I'm not sure it'll be available for the next release...

Sunday, January 21, 2007

Long time...

Yeap, it's been quite some time since my last post, but I've been busy ;-)

Right now, the aim is getting things ready for the pydev 1.2.6 release (which hopefully will happen this week)...

Pydev already had many fixes, in the package explorer, templates, changes from Thread to the Job API, etc, etc.

And in pydev extensions, the next version will implement the refactoring for rename... Still the first release, so, it should still be regarded as beta, but it already features a preview for the refactorings, undo/redo (both courtesy of the Eclipse framework), and should work for most rename refactoring cases, such as classes, methods, attributes, parameters, modules (although you have to make this refactoring through some import, because it is still not integrated in the package explorer)...

As it is beta, please take some time and review the changes it will do... and report if you think it forgot to rename something or renamed more than it should -- but have in mind that you're refactoring python, and as it is very dynamic, some analysis might not even be possible (but should probably be ok for 90% of the cases).

And another nice thing is that there is a really cool project called PEPTIC that will add other refactorings to pydev, and should shortly replace BRM as the default refactoring engine in pydev (that will probably happen around march)...