GeoTools : 3 November 15th

Conversation with #geotools at 2004-11-15 11:43:18 on (irc)
(11:43:18) : The topic for #geotools is:
(11:43:25) IanS: yes, mostly, and I,er, mostly agree
(11:43:31) cholmes: Cool IanS, would be great to have your thoughts, since all the FeatureCollection in the Collection framework was your idea, we're just trying to bring it front and center.
(11:44:34) IanS: i agree that familiarity with api is a good thing, the random access points from martin were also good
(11:44:51) IanS: maybe to start with, limit it to in memory...
(11:45:11) cholmes: limiting random access to in memory?
(11:45:30) IanS: no, the collection itself
(11:46:10) IanS: mostly with regards to cleanup of connections/file streams if the collection is not an in-memory collection
(11:46:22) polio: mmm, will be hard to get buy in (atleast from most of the guys here) for an in-memory solution ... cause it wouldn't really change much
(11:46:54) IanS: to begin with, geotools was all in-memory and you're here, right?
(11:47:07) jmacgill: The question is what does the API need to look like (implemenation for a stream based appraoch may take a while to put in place) but we need the api now
(11:47:19) IanS: agreed
(11:47:40) jmacgill: so, if we need things like dispose, or close(), then where should they be
(11:47:45) polio: yah, was just concerned we might be taking a step backwards
(11:47:54) cholmes: Well, actually getting geotools out of all in memory was the first thing refractions worked on...
(11:48:02) IanS: i suppose the addition of a dispose method to a collection could work, along with a finalize() cleaner or perhaps a PhantomReference quere cleaner
(11:48:12) cholmes: And it's not in memory now, with feature results and feature readers.
(11:48:43) cholmes: I don't think it's needed in the collection, just in the FeatureIterator...
(11:48:55) polio: chris: my point, so was just concerned we may be going backwards there ... was hoping we would make the api assuming out-of memory collections
(11:49:09) pramsey entered the room.
(11:49:13) IanS: i know, my point was made clearer by james, the fully-featured bits may not be available immediately, but folks will still use it
(11:49:31) cholmes: Oh ok, cool.
(11:51:07) jmacgill: I'm not a fan of dispose methods... but if fianlize is not guranteed then I guess we have to
(11:51:31) cholmes: But yeah, I don't believe the collection needs a dispose method, since it would replace our FeatureResults, which does not have a close method. It just needs it in the iterator.
(11:51:32) polio: chris, you mentioned a close method in the featureIterator, does that imply that you would prefer not to use the Itertor interface as a return type?
(11:51:53) cholmes: I never got a chance to respond to the emails, but yes, I am very much for a close in FeatureIterator.
(11:52:22) polio: no, dispose is not guaranteed (also causes some issues for large memory systems, small max connections)
(11:52:23) cholmes: I feel like there were some original motivations for having FeatureIterator not implement Iterator...
(11:53:15) cholmes: Probably not wanting to support remove() ?
(11:53:17) polio: was asking because it is a step away from the collections api (which appears to be our design basis)
(11:53:48) polio: wanted to support remove on writable feature collections
(11:53:53) cholmes: True, but we do need a close somewhere, no? I think that parting at that point would be the best way.
(11:54:17) cholmes: Because it also has a nice little improvement on iterator - don't need to cast to Features...
(11:54:25) jmacgill: we could extend Iterator.. but by returning FeatureIterator the method would be accessable
(11:54:25) cholmes: As iterator just returns objects.
(11:54:31) polio: I would have prefered at the collection level, as you need the connection for random access get(x)
(11:54:54) cholmes: Come again? What do you mean?
(11:54:56) polio: and the main point I see for close is to allow for connection management
(11:55:51) jmacgill: collections define the params for a connection, they are not an actual connection
(11:55:54) cholmes: (note for random access in jdbc I think you're going to have to change how it issues its calls, and you're going to get back the result sets all in memory, though I could be wrong)
(11:55:55) polio: FeatureCollection.get(int i) or FeatureCollection.get(Fid) needs a collection presumably, and this does not require the iterator() method to be called
(11:56:06) cholmes: Right.
(11:56:22) polio: so the FeatureCollection needs a connection ... which needs to be closed
(11:56:56) cholmes: Ah, I got you...hmmm...
(11:57:50) cholmes: I feel like someone's probably done this before, a sql backed Java collection - I wonder if we could borrow someone's design.
(11:58:09) jmacgill: FeatureCollection does not have a getX, but List does
(11:58:40) polio: yes, was assuming we are talking about the proposed FeatureCollection
(11:58:41) cholmes: So perhaps FeatureList has a close?
(11:59:18) cholmes: Interesting note from the Collections design docs faq: Why doesn't Collection extend Cloneable and Serializable?
(11:59:27) cholmes: Many Collection implementations (including all of the ones provided by the JDK) will have a public clone method, but it would be mistake to require it of all Collections. For example, what does it mean to clone a Collection that's backed by a terabyte SQL database? Should the method call cause the company to requisition a new disk farm? Similar arguments hold for serializable.
(12:00:32) polio: james: yes, but collection does have contains(Object), remove(Object) and add(Object) all of which may require a connection
(12:00:59) polio: it also has toArray ... which i'm guessing is optional to implement
(12:02:10) cholmes: remove and add are also optional
(12:02:15) jmacgill: if we add dispose to the FC can we cause any existing Iterators to die?
(12:02:20) cholmes: and contains you can use an iterator
(12:02:50) polio: yes, that would just require some house keeping on the FC's part
(12:04:07) jmacgill: I should say rather, it is acceptable behavour for existing Iterators to just stop working.
(12:04:34) polio: yes, the hasNext() would return false ...
(12:05:02) jpmeagher: The ConcurrentModificationException could be thrown in this case
(12:06:24) jmacgill: tbat makes a lot of sense
(12:06:50) jmacgill: sooo, the FC would/could act as a connection pool
(12:06:51) cholmes: Yeah, everyone should read over the collection design notes before we move on this, there's all kinds of good advice.
(12:06:56) cholmes: Like: Attempting to add an ineligible element throws an unchecked exception, typically NullPointerException or ClassCastException.
(12:07:22) cholmes: So when a Feature doesn't validate according to the FeatureType it should do that (or perhaps a specialized ClasscastException)
(12:07:46) jmacgill: I thought it trhoew an illigalargument exception?
(12:08:50) cholmes: Um, I'm just cutting and pasting the docs...
(12:10:26) jmacgill: so, we think we can have a FeatureCollection that implements collection but also has a dispose method
(12:11:02) jmacgill: ok, what about transactions
(12:11:32) polio: well, I have basically the same arguement to place the get/set methods in featurecollections
(12:12:13) cholmes: I'm fine with that - I personally don't see a huge need to implement this right now. I would pretty much always like featureStores around, I see the get/set as another way in, not the main way.
(12:13:12) jmacgill: I was mostly concerned about what the API that was presented to GeoAPI looked like
(12:13:38) polio: same here
(12:13:48) jmacgill: I also want to try and make this work as I think it will be far easier to work with
(12:15:12) polio: I see this as a goal for the future, obviously it won't happen overnight. I think we should figure out the optimal solution, and then work on a timeline for the transition
(12:16:11) cholmes: I'd actually sorta like to see the basic changes soon, as there is more we need to improve...
(12:16:45) jmacgill: what do we need as a bare minnimum then... a dispose method
(12:16:57) jmacgill: and a mechanism for obtaining a FC from a DS that we are happy with
(12:17:55) jmacgill: having the DS construct the FC is handy as it allows each DS to create an optimal FC implementation.
(12:18:05) polio: yes, and a consensus as to what the minimum level of random access support should be, along with a plan for extended access operations (indexes?)
(12:18:49) cholmes_ entered the room.
(12:19:26) polio: I see a collections based featureCollection as replacing the Featurestore ... apis in the future
(12:19:28) cholmes_: sorry, looks like I hit a bump, did you guys get two messages from me, about doing the basic changes soon?
(12:19:37) polio: nope just one (smile)
(12:19:57) cholmes_: Second was: The list stuff and transactions could wait for a bit, but change over FeatureStores to returning FeatureCollections relatively soon.
(12:20:34) jmacgill: FCs come from FeatureStores not DataStores?
(12:20:40) polio: mmm, what is the significant difference between featureSources and FeatureCollection (with the collection api)?
(12:21:07) polio: I would have expected the FC's to come from the DS
(12:22:50) jmacgill: me too, I think a FeatureSource just adds to the complication
(12:24:25) jmacgill: I think we may have lost Chris
(12:24:53) polio: should we come to a concensus?
(12:24:53) polio: DS produces FC's
(12:24:53) polio: FC's have a dispose() method
(12:24:53) polio: FC's have get/set transactions?
(12:25:04) polio: hmm, will have to wait for chris then
(12:26:19) jmacgill: ok, for now then I'll move on to the status of 2.0.0
(12:26:35) jmacgill: I have finaly tracked down a build bug that was holding me back
(12:27:04) jmacgill: the build script that comes with the 2.0 release was only working on Java 5 (1.5)
(12:27:18) jmacgill: which is why it worked for some people and not others
(12:27:30) polio: I havn't tested yet (... starting now)
(12:28:00) jmacgill: I've put up a new sanity check release which I will turn into 2.0-rc2 if it seems even remotly sane (i.e. builds for at least one person that it did not build for before)
(12:29:54) jmacgill: if all goes well I'll release 2.0 in a week
(12:32:28) rschulz: I will test it again tonight (java 1.5)
(12:32:51) jmacgill: Chris has gone offline. I suspect he has that rogue process again
(12:32:55) jmacgill: he is proably rebooting
(12:33:32) polio: do we need a java 1.4 windows test?
(12:34:21) jmacgill: would be very handy yes
(12:34:32) polio: will see what i can do there
(12:34:44) polio: (can easily test 1.4 linux)
(12:35:02) cholmes__ entered the room.
(12:35:14) cholmes__: sorry guys, third world internet can get you down sometimes...
(12:35:19) rschulz: I may also have time to try 1.4 on mac osX
(12:35:23) cholmes__: could someone email me the log so I can catch up?
(12:35:23) polio: 3 chris's at once (smile)
(12:35:50) cholmes__: two of them living in cyber limbo...
(12:38:26) jmacgill: I was just updating on the 2.0 release whilst you were gone
(12:38:46) jmacgill: can we go back to FC
(12:39:05) jmacgill: the main question was that we though they should come directly from a DS and not a FS
(12:39:23) cholmes left the room (quit: Read error: 110 (Connection timed out)).
(12:39:30) polio: doh
(12:39:37) cholmes__: I'm still here.
(12:40:05) cholmes__: I don't think they should, did you get any of my points on that?
(12:40:21) cholmes__: I am far less convinced on featurecollections replacing FeatureStore.
(12:40:25) cholmes__: And that's way further than I seek to go.
(12:40:31) cholmes__: Though I could be convinced at some point - but as a WFS I really like the WFS interface, it makes my job a lot easier.
(12:40:36) cholmes__: Having it all modeled on how I actually want to work, and I do think it's a good model to work in, if you're doing transactions and locking.
(12:41:27) polio: mmm, so chris could you outline what you get from a FS that the proposed FC does not provide?
(12:41:46) polio: (would provide a good starting point for discution)
(12:41:49) cholmes__: In the next couple months I'm looking to do even more with FeatureStores, to allow FeatureViews, decoupling the view from the actual data.
(12:42:03) cholmes__: Mostly mental division between concepts.
(12:42:11) cholmes__: We've got a lot going on...
(12:42:20) cholmes__: I don't really want to issue Queries against FeautreCollections
(12:42:38) cholmes__: I totally see that you could, but I would rather think of the FeatureSource as what I issue queries against.
(12:42:40) cholmes_ left the room (quit: Read error: 110 (Connection timed out)).
(12:42:47) cholmes__: (still here)
(12:43:05) jmacgill: no, nore should you. I assumed the FC would be the result of a Query agsinst a DS
(12:43:17) cholmes__: Especially when I start to make FeatureSources that are based on Joins, and that do complex mappings from other FeatureIterators.
(12:43:53) cholmes__: A datastore is too big of a thing, it has a number of different featureTypes...
(12:44:10) cholmes__: i want to be able to do my work knowing what I'm working against - a single table or file.
(12:44:10) polio: but, wouldn't a query include the featureType?
(12:45:02) cholmes__: Yes, it would, for me it's a matter of keeping things straight in my head. And also management of the whole DataRepository. That in GeoServer each FeatureType that I give information on has a set FeatureSource/store.
(12:45:20) cholmes__: I completely agree that you could do everything with FeatureCollection...
(12:45:25) cholmes__: For me it's just going too far right now.
(12:45:44) polio: I guess what I most want to see is the end result
(12:46:10) polio: basically I think we really should propose the ideal to geoapi
(12:46:16) cholmes__: I'd like to actually implement this idea first, and then be able to tackle Joins and Views and complex mappings in the DataStore framework, and then maybe consider moving it all to FeatureCollection.
(12:46:29) polio: and slowly move towards that with the code base (ie not all at once)
(12:46:53) cholmes__: I think that'd be great, but i don't know that we have the time to really design it all right at the moment...
(12:47:21) cholmes__: I certainly don't have the resources to a bunch of my thoughts into things that are so far in the future...
(12:47:42) jmacgill: depends where we draw the line in terms of API
(12:47:55) Martin_ entered the room.
(12:47:59) jmacgill: if GeoAPI just woried about FC and not wwhere they came from
(12:49:00) jeichar entered the room.
(12:49:06) polio: yah that would be a good start
(12:49:34) jmacgill: we have aggreed on some issues, like the need for a dispose at the FC level
(12:49:43) jmacgill: and a get/set transaction method
(12:50:13) jmacgill: I see a need to tie the FC back to the DS, but I think thats an implementation issue
(12:50:53) polio: I could be convienced that FS.getCollection(Query) is really just a convinience method ...
(12:51:20) jmacgill: Can we go as far as to include an interface for a 'thing' from which you can obtain a FC
(12:51:23) polio: but this does leave us with a complex api
(12:52:04) polio: I think that they are looking for this too ... atleast the first go included such an interface
(12:53:06) jmacgill: humm, what does FS have now?
(12:53:52) cholmes__: I personally would really prefer we implement this and work with it a bit before going nuts with all the fun implications you can have with it.
(12:54:15) polio: well, currently the execution for data can go DS->FS->FResults->FeatureReader ... bit too much overhead for me
(12:54:24) cholmes__: Because I need to start tackling joins, for real, not theoretically, and it's much easier for me to think in the current feature source paradigm, as I know it a lot better.
(12:55:43) jmacgill: FS has nothing in it of any note. Now, subclasses, like JDBCFeatureSource do (like getTransaction) but thats hidden
(12:56:01) cholmes__: I personally think those are valuable divisions, as to what each actually is.
(12:56:18) jmacgill: AbstractFeatureSource has getTransaction !
(12:56:48) polio: I just see it as a DS wrapper with a query and transaction ...
(12:56:58) polio: believe getTransaction is in the api
(12:57:20) jmacgill:
(12:57:28) jmacgill: nope
(12:58:18) jmacgill: Ah, but FeatureStore does. OK, I'm officaly confused all over again
(12:58:24) polio: you are correct ... it's in featurestore
(12:59:17) polio: (was really thinking of FeatureCollection as equivalent to a featurestore as both can be read/write)
(13:00:46) polio: gt2:main cleaned, compiled, tested, INSTALLED 1237
(13:00:46) polio: gt2:arcsde cleaned, compiled, tested, INSTALLED 1237
(13:00:46) polio: gt2:geomedia cleaned, compiled, tested, INSTALLED 1237
(13:00:46) polio: gt2:gtopo30 cleaned, compiled, tested, INSTALLED 1237
(13:00:46) polio: gt2:mysql cleaned, compiled, tested, INSTALLED 1238
(13:00:46) polio: gt2:shapefile cleaned, compiled, tested, INSTALLED 1238
(13:00:46) polio: gt2:postgis cleaned, compiled, tested, INSTALLED 1239
(13:00:46) polio: gt2:tiger cleaned, compiled, tested, INSTALLED 1239
(13:00:46) polio: gt2:graph cleaned, compiled, tested, INSTALLED 1239
(13:00:46) polio: gt2:j2se-demos cleaned, compiled, tested, INSTALLED 1239
(13:00:46) polio: gt2:reprojection cleaned, compiled, tested, INSTALLED 1239
(13:00:46) polio: gt2:validation cleaned, compiled, tested, INSTALLED 1239
(13:01:00) polio: seems sane to me (smile)
(13:01:08) jmacgill: (smile)
(13:01:15) cholmes__: where can I get it? I can try it out...
(13:01:38) cholmes__: Also I just put an oracle fix into svn, can you try to get that in?
(13:02:00) jmacgill:
(13:02:12) jmacgill: yes, I'll cut a new one
(13:02:20) jmacgill: will probably have vpf in as well
(13:02:46) jpmeagher: cool
(13:03:25) jmacgill: the role back happend just in time (smile)
(13:05:57) cholmes__: Ok, so if FeatureSource really is just a convenince, can you keep it in for me? I like the convenience of it. And then you can replace DataStore.getReader(Query, transaction) with DataStore.getFeatureCollection(query, transaction)
(13:07:17) jmacgill: looks good to me. We can (and proably need) to keep old stuff around for quite a while
(13:07:43) polio: yes, but can we also deprecate getFeatureSource?
(13:08:30) cholmes__: If you're fine with geoserver using a deprecated API for a long time...
(13:08:47) jmacgill: I'd say leave it undepricated for now.
(13:08:58) jmacgill: we can depricate it if/when Chris is happy with an alternative
(13:09:06) cholmes__: I'd prefer you to prove to me how much better you new api is working
(13:09:08) cholmes__: exactly
(13:09:21) cholmes__: and how it can more easily solve my issues, like joins
(13:10:14) jmacgill: but life for end users will be eaiser as we can point them to the GeoAPI spec
(13:10:18) polio: yes, understand your point ... was only hoping to make the api clearer to new users, kinda point then towards the simplest solution for them
(13:10:47) cholmes__: I'm likely going to implement joins with a feature source like construct, but if all goes well you should be able to flip it around to your way as well, it's all just different ways of looking at the same shit. Indeed I can play mental games with stuff like FeatureCollections.getView(FeatureCollection in, MappingStrategy mapping, FeatureType out). I just need to conceptually work in the FeatureSource space for now.
(13:11:04) cholmes__: Well, if it's cleaner to new users it should naturally point in that direction (wink)
(13:11:14) jmacgill: how well does Query handle FeatureTypes?
(13:11:35) cholmes__: Huh?
(13:12:03) polio: has the typeName, namespace, and property names
(13:12:12) polio: so about 1/2 of a featuretype
(13:12:17) jyutzler left the room (quit: "ChatZilla 0.9.61 1.7.3/20040910").
(13:12:24) cholmes__: I also am thinking about scaling back Queries a bit, so that they never really modify the FeatureSource...
(13:12:38) cholmes__: I'd also like to see propertyNames not be in different FeatureTypes.
(13:12:53) cholmes__: I think a feature with just a few properties should be able to validate against the full FeatureType
(13:12:55) jpmeagher left the room.
(13:12:59) cholmes__: instead of our wierd retyping going on now.
(13:13:24) cholmes__: I'd like retyping to happen when we are actually changing the FeatureType, renaming attributes or doing new mappings.
(13:13:34) polio: well, shouldn't the inheritance take care of some of that (I do realize it hasn't fully propagated yet)
(13:13:49) jmacgill: The question relates to geting a FC out of a DS using ds.getFeatureCollection(Query, Transaction)
(13:13:52) rschulz left the room (quit: "Leaving").
(13:14:05) cholmes__: And in that vien I think the coordinateSystem over-ride should not be in Query (though reprojection should)
(13:14:12) jmacgill: this works if Query can spefy the FeatureType well enough
(13:14:29) cholmes__: And indeed if it could establish all the new mappings that I want to do.
(13:15:05) polio: I had a proposal that had query attributes defined as ogc expressions ...
(13:15:08) cholmes__: That's why I see something like FeatureSource as valuable right now, because I can generate a new one given an existing FeatureSource and some new mappings - create virtual FeatureSources (FeatureViews)
(13:15:19) cholmes__: Yeah, I think that's overloading Query too much.
(13:15:34) cholmes__: Makes constructing a query to get your features too hard to use, not intuitive at all.
(13:15:38) polio: based the idea off an sql statement
(13:15:47) cholmes__: A query should be a query, not a redefinition.
(13:15:56) cholmes_: I like the idea for _defining a new feature source.
(13:16:09) polio: yet sql lets you redefine ...
(13:16:30) polio: and is the predominant data query language
(13:16:32) cholmes__: Yeah, I know all this is possible both ways, for me it's just going too far right now.
(13:16:46) cholmes__: And I can be convinced in the future, if you're able to come up with compelling implementations for what you want.
(13:17:27) cholmes__: But an idea 'based off of a sql statement' is not a sql statement
(13:17:34) cholmes__: and thus is not the predominant data query language
(13:18:35) polio: ok, but you were claiming the redefining stuff should not be in the query ... and I used sql as proof that it would not be a new convept
(13:18:38) polio: *concept
(13:19:17) cholmes__: It wouldn't be a new concept, but it would be a very heavy concept in something I think is already getting to heavy.
(13:20:07) cholmes__: That said, you're getting into my thoughts that aren't nearly as well fleshed out.
(13:20:19) cholmes__: And a good review from you all once I do flesh them out will be good.
(13:20:31) cholmes__: I'm just asking to keep FeatureSource around for a bit to help me flesh them out.
(13:20:40) polio: cool, was trying to figure out what construct you had in mind
(13:21:27) polio: yup, don't think we are disagreeing with it's existance (perhaps a bit on it's public visibility ... but that only a small issue)
(13:21:29) cholmes_: Oh, it would more be to seperate out the _definition of the remapping and redefining into the FeatureSource, just to keep them conceptually seperate.
(13:22:00) cholmes__: And in many ways that may just be a crutch for me to figure it out, and could go back once I get it.
(13:22:33) cholmes__: But I've just been realizing implementing joins for real is a much, much more involved task than simply thinking of a nice api for it.
(13:23:53) cholmes__: So I would just say to start that it stays in a syntactically different structure (the redefinition that is) - which for me would correspond to what a WFS user actually issues queries against (which is a FeatureSource)
(13:24:26) cholmes__: I'll try to write things up soon, these thoughts are still in pretty early stages.
(13:24:59) polio: cool, we did some thinking on this, and figured that the feature schema could be used substantially for this
(13:25:28) polio: which coincides with your plans too
(13:26:14) cholmes__: could you explain that a bit more?
(13:26:23) cholmes__: feature schema being the FeatureType?
(13:26:25) polio: as an aside, we may have to revisit FT definition, as it could well break
(13:26:39) cholmes__: We will definitely have to revisit FT definition
(13:26:50) polio: oh, a gml schema does not require all the attributes to be in the same namespace ...
(13:27:13) polio: so you can represent derived types
(13:27:46) polio: you can also use xlink in gml to represent nested (posibly joined ...) feature sets
(13:28:07) polio: but then the parser writers have alot of nasty work to do (sad)
(13:28:23) cholmes__: And you're thinking of moving this to the FeatureType api?
(13:29:08) cholmes__: Oh yeah, I did like the thoughts that a FeatureCollection is a Feature, I need to pursue it a bit more, but I think it could nicely replace my MultiAttributeType stuff...
(13:29:09) polio: well, I don't see how you can merge two namespaces into one feature type without something along those lines
(13:29:53) polio: presumably you will want to maintain the inheritance of the attributes through the namespace declarations
(13:30:04) cholmes__: Instead of a MultiAttributeType you just have a FeatureCollection, and an attribute is a feature, and can do validation and all that...
(13:30:17) cholmes__: Oh, the stuff I'm looking to do isn't so much involved in that...
(13:30:23) cholmes__: I don't care so much about joining namespaces.
(13:30:46) cholmes__: I just want to be able to combine things on the backend.
(13:31:27) polio: yes, but people could well want that in the future (and it isn't too unreasonable) .. just requires a few back-pointers
(13:32:18) cholmes__: True, but the amount of gain it gives you vs. the distance I need to go where people would want it is super substantial.
(13:32:24) cholmes__: Once I get there I'll hit you up... (wink)
(13:32:37) polio: yup
(13:32:47) polio: just doing a brain dump on FT
(13:33:18) cholmes__: Cool.
(13:34:43) jmacgill: I have to run
(13:34:53) polio: yah, I should too
(13:36:06) jmacgill: this has been a good session
(13:36:25) jmacgill: I'll try and formalize some of it if I can soon
(13:37:24) polio: cool, I can send you the logs (for the portion i was here) if you need them
(13:37:28) polio: missed the first 10 min
(13:37:39) jmacgill: please do