Date   

How to deal with deprecated / unsupported GDAL drivers

René Buffat
 

GDAL now maintains a separate project with unsupported drivers: https://github.com/OSGeo/gdal-extra-drivers

With the new release of GDAL 3.3.0 more drivers are marked as deprecated: https://github.com/OSGeo/gdal/releases/tag/v3.3.0

 

The following drivers included in Fiona are already moved to gdal-extra-drivers with GDAL 3.3: AeronavFAA, BNA, SEGY, SUA.

With GDAL 3.5, the following drivers will be removed from GDAL (and probably moved to gdal-extra-drivers): ARCGEN, GTM=GPSTrackMaker?

 

I assume most of these drivers can be considered "exotic". Thus I would suggest removing them from Fiona supported driver list with the first release of Fiona 1.9.x. 

Also, this would be a good occasion to review the list of GDALs currently supported drivers to check if new drivers should be supported.  

 


Re: Fiona project status report

Alan Snow
 

Hi Sean,

Thanks for your work on Fiona and for the status update!

Here are some thoughts I have:
1. For maintainability reasons, I am wondering if fiona should bump up the pin for the supported GDAL version to 3+? 3.0.4 is the version for Ubuntu 20.04 LTS (https://packages.ubuntu.com/search?keywords=gdal).
2. GitHub Actions is pretty nice. I recommend looking into it as an alternative to Travis CI.
Potentially useful references:
- https://github.com/pyproj4/pyproj/tree/master/.github/workflows
- https://github.com/pyproj4/pyproj/issues/743
3. It wouldn't hurt to get started on transitioning the wheels with the existing infrastructure while it is still there. It would likely remove some of the complexity in the transition. But, it all depends on when you get time to work on it.


Fiona project status report

Sean Gillies
 

Hi all,

I'm writing this little note to summarize where the project stands at the end of 2020, a year of uncertainty and ambiguity.

First, the status of the code base. To be frank, the rewrite of fiona's data model for 1.9/2.0 is stalled. The time I had at my day job to start the work has been eaten up by other projects in 2020. I honestly don't know when I'll be able to reverse this trend, but I am working on it. We've continued to make 1.8.x releases to fix bugs and have merged some commits into fiona's main branch, the one we have planned to release 2.0.0 from. The main branch has not merged 1.9 and has not merged maint-1.8 in some time. We have a PR that will address that last issue, and at least we will have maint-1.8 and the master branch aligned again.

Now, the status of CI and build infrastructure. As you know, Travis CI is cutting support for open source projects without a subscription. This puts fiona's CI and wheel builds in some jeopardy. At some point we will run out of credit and Travis will not build the project or its wheels. In the short term we can pare down our build matrix and petition Travis CI for more credits. I'll take the lead on the latter. In the long term, we need to explore other CI options and other wheel building infra like cibuildwheel.

Speaking of wheels, the version of GDAL we include in wheels is becoming unmaintainably old and harder to patch. Switching to GDAL 3.2 and PROJ 7.2 is a priority for early 2021. As is adding wheels for Python 3.9 and 3.10. I'm not sure whether this work is something to be done within our existing, deprecated infra, or if we should wait until we switch. I'd love to hear from other perspectives on this.

Thank you, contributors. René Buffat, Alan Snow, and Mike Taves have done especially important work this year. Fiona wouldn't be where it is without your effort!

Sincerely,

--
Sean Gillies


Re: Performance regression: fiona.transform much slower since GDAL 3

stefan.brand@...
 

Hi Sean,

thanks for putting my mail in the right place!

The sqlite version should not be a problem as we have sqlite 3.33
installed.

It seems that the performance issue indeed was with initializing the
transformer over and over again. I get much better results with
pyproj: https://gis.stackexchange.com/a/376247/122597

I will test this fix in our real-life code and get back to you if I run
into troubles.

Thank you!
Best, Stefan

On Fri, 2020-10-09 at 12:24 -0600, Sean Gillies wrote:
Hi Stefan,

I'm cross-posting this over to main@fiona.groups.io. The dev list is
for project planning and design discussions.

Is it possible you are running into the issue with older sqlite
described in https://github.com/OSGeo/PROJ/issues/1718?

On Fri, Oct 9, 2020 at 4:53 AM <stefan.brand@eox.at> wrote:
Hello everyone,


I kindly request your comments/answers to this gis.SE question,
where I
have detailed my issue:
https://gis.stackexchange.com/q/376211/122597

If you do not have an account there, but still know an answer,
please
contact me directly via e-mail.

Thank you for your attention!


Best, Stefan







Re: Performance regression: fiona.transform much slower since GDAL 3

Sean Gillies
 

Hi Stefan,

I'm cross-posting this over to main@fiona.groups.io. The dev list is for project planning and design discussions.

Is it possible you are running into the issue with older sqlite described in https://github.com/OSGeo/PROJ/issues/1718?

On Fri, Oct 9, 2020 at 4:53 AM <stefan.brand@...> wrote:
Hello everyone,


I kindly request your comments/answers to this gis.SE question, where I
have detailed my issue: https://gis.stackexchange.com/q/376211/122597

If you do not have an account there, but still know an answer, please
contact me directly via e-mail.

Thank you for your attention!


Best, Stefan









--
Sean Gillies


Performance regression: fiona.transform much slower since GDAL 3

stefan.brand@...
 

Hello everyone,


I kindly request your comments/answers to this gis.SE question, where I
have detailed my issue: https://gis.stackexchange.com/q/376211/122597

If you do not have an account there, but still know an answer, please
contact me directly via e-mail.

Thank you for your attention!


Best, Stefan


RFC: Roadmap for Shapely 2.0

Joris Van den Bossche
 

Dear list,

This is not directly a Fiona-related message, but I assume there might be some of you that are also Shapely users. For those, I want to get across a message about a proposal for Shapely 2.0 (and if not, you can skip the rest of the message ;)).

There is a proposal for some significant changes in Shapely: "RFC 1: Roadmap for Shapely 2.0" (https://github.com/shapely/shapely-rfc/pull/1).

Background: we have been working on a new set of Python bindings to GEOS in the PyGEOS package (https://github.com/pygeos/pygeos/). The initial focus was to make it easier and more performant to work with arrays of geometries (where Shapely is now focused on scalar geometries). See https://caspervdw.github.io/Introducing-Pygeos/ for some more details. While this separate package made it easier to experiment (and which we are still doing), on the long term, it's not ideal to have two incompatible GEOS wrappers in the Python ecosystem. Therefore, the proposal linked above is about upstreaming this work into Shapely, along with some API changes.

Given that this are some profound changes to Shapely, feedback from Shapely users would be very much appreciated! 

Best,
Joris



Support for measured geometries in Fiona 2.0

Marouane
 

Hello all,

I am new to the Dev group, I have previously worked a little on a fork of Fiona 1.8  to add support for the Polyline M geometries.
As I understand, measurement values support is a big change and there are a lot of things to consider before diving in. But I hope you guys agree that it's an important feature for the upcoming Fiona release. and it would allow extending M-values support to modules that use Fiona for read/write purposes such as Geopandas.

So what are your thoughts on this ? Suggestions ?


GDAL 3 support in PR 804

Sean Gillies
 

Hi all,

Sorry for the lack of notice, but I had a rare opportunity to stay up late programming last night and threw together a PR to get us out of GDAL 2/3 purgatory: https://github.com/Toblerity/Fiona/pull/804.

Grateful for reviews,

--
Sean Gillies


Re: Amending RFC 1 to make GDAL 2.5 the baseline for Fiona 2.0

Alan Snow
 

Brain dump continued:
Another option would be to have pre-releases of Fiona 2.0 like what happened for rasterio 1.0 where users have to explicitly pin the pre versions. This would allow users to opt-in to the changes until GDAL 2.5 becomes more mainstream. Additionally, this would alleviate the burden of supporting older versions of GDAL in Fiona. But, depending on the timeline of when you want to release Fiona 2.0, this may or may not be desirable.


Re: Amending RFC 1 to make GDAL 2.5 the baseline for Fiona 2.0

Alan Snow
 

Hi Sean,

I think a major version bump in Fiona would be a great opportunity to introduce breaking changes and new features to help guide the community to new (and hopefully improved) standards. But, I also understand the desire for backwards compatibility and to help users migrate over. One way to help users along with the migration would be to go the direction you have outlined, carrying along with it the burden and limitations of supporting the old and the new versions into the next major version. Another option would be to have a 1.9.x branch of Fiona to support GDAL 2.2-2.4 and backport new features developed in Fiona 2.x (GDAL 2.5+) where possible (also a burden). This would enable Fiona 2.x to fully migrate over to the new features in the GDAL 2.5 (which could arguably be a 3.0 release). There are also other options I am sure, just not currently in my brain :). That being said, either direction has its challenges, so I am supportive of either direction :+1:.

Best,
Alan


Re: Amending RFC 1 to make GDAL 2.5 the baseline for Fiona 2.0

Alan Snow
 

Hi Howard,

I am happy to hear that pyproj works well for you! The point cloud format you linked to looks interesting - glad you posted it.

Best,
Alan


Re: Amending RFC 1 to make GDAL 2.5 the baseline for Fiona 2.0

Sean Gillies
 

Hi Alan,

I don't think we should switch to WKT2 yet. The goal here is to release a version of Fiona that works with GDAL 2.5 so the upgrade paths aren't blocked, even if we don't use all the new features of 2.5.


On Tue, Apr 23, 2019 at 6:20 AM Alan Snow <alansnow21@...> wrote:
Going to 2.5 sounds like a great way to go! I think that there are some significant changes there that make that a good choice. But, I think testing should be done with the earlier versions before committing to supporting them. There have been changes that might not backport well. For example, if you want to go with WKT2 as the new default, will it still work in the previous versions?



--
Sean Gillies


Re: Amending RFC 1 to make GDAL 2.5 the baseline for Fiona 2.0

Sean Gillies
 


On Mon, Apr 22, 2019 at 10:20 PM Sean Gillies via Groups.Io <sean.gillies=gmail.com@groups.io> wrote:
Hi all,

The implications of the changes in GDAL 2.5 have finally dawned on me. I think we need to make 2.5 the baseline version of GDAL for Fiona 2.0 and then add shims to provide what is missing in GDAL versions 2.2-2.4: specifically the data-CRS axis order mapping.

Unless anyone objects, I will amend the RFC to state clearly that GDAL 2.5 will be the new baseline version.

--
Sean Gillies



--
Sean Gillies


Re: Amending RFC 1 to make GDAL 2.5 the baseline for Fiona 2.0

Howard Butler
 

We are strongly considering requiring GDAL 2.5+ when we release PDAL 2.0 this summer for similar reasons. PDAL and libLAS were using some older libgeotiff methods like OSRFixup that have since been removed, which makes for a few annoying #ifdefs. The larger challenge is coordinate system transformations are likely to behave differently at the edges of functionality depending on which version of the GDAL/PROJ stack that is linked if we supported both, and that is combined with our desire to hurry up already and start using the clean concourses of the new PROJ/GDAL architecture. 

PROJ had been targeting removal of proj_api.h to attempt to bring forward many of the laggards that have been using PROJ internals at its PROJ 7 release, but I think this is going to have to be at least delayed. The impacts for the packagers are still going to be quite punishing for a while on that topic. PROJ moved glacially forever, and people's attention has not caught up with its new pace.

Alan, as an aside, I finally got to do some Python while implementing a library for our Entwine Point Tile [1] point cloud format, and I was using the new pyproj with PROJ 6. Everything Just Worked(tm) pleasantly. A long time ago when I was developing lots of Python stuff, having a nicely organized library with those capabilities was the dream. Nice work!

Howard


On Apr 23, 2019, at 7:20 AM, Alan Snow <alansnow21@...> wrote:

Going to 2.5 sounds like a great way to go! I think that there are some significant changes there that make that a good choice. But, I think testing should be done with the earlier versions before committing to supporting them. There have been changes that might not backport well. For example, if you want to go with WKT2 as the new default, will it still work in the previous versions?


Re: Amending RFC 1 to make GDAL 2.5 the baseline for Fiona 2.0

Alan Snow
 

Going to 2.5 sounds like a great way to go! I think that there are some significant changes there that make that a good choice. But, I think testing should be done with the earlier versions before committing to supporting them. There have been changes that might not backport well. For example, if you want to go with WKT2 as the new default, will it still work in the previous versions?


Re: Amending RFC 1 to make GDAL 2.5 the baseline for Fiona 2.0

Joshua Arnott
 

I hadn't been following along with the GDAL 2.5 changes either. I agree with the proposal to make GDAL 2.5 the baseline for Fiona 2.0, with support for old versions (hopefully not too difficult!) added via the shims.

For those just catching up, I found this link helpful:

https://trac.osgeo.org/gdal/wiki/rfc73_proj6_wkt2_srsbarn


Re: Amending RFC 1 to make GDAL 2.5 the baseline for Fiona 2.0

Joris Van den Bossche
 

Sure!

Op di 23 apr. 2019 om 06:20 schreef Sean Gillies <sean.gillies@...>:

Hi all,

The implications of the changes in GDAL 2.5 have finally dawned on me. I think we need to make 2.5 the baseline version of GDAL for Fiona 2.0 and then add shims to provide what is missing in GDAL versions 2.2-2.4: specifically the data-CRS axis order mapping.

Unless anyone objects, I will amend the RFC to state clearly that GDAL 2.5 will be the new baseline version.

--
Sean Gillies


Amending RFC 1 to make GDAL 2.5 the baseline for Fiona 2.0

Sean Gillies
 

Hi all,

The implications of the changes in GDAL 2.5 have finally dawned on me. I think we need to make 2.5 the baseline version of GDAL for Fiona 2.0 and then add shims to provide what is missing in GDAL versions 2.2-2.4: specifically the data-CRS axis order mapping.

Unless anyone objects, I will amend the RFC to state clearly that GDAL 2.5 will be the new baseline version.

--
Sean Gillies


Re: A repo and template for Fiona requests for comments (RFC)

Sean Gillies
 

Hi all,

RFC 1, Fiona 2.0 changes, is finished.


Thanks for the feedback. It's much better now than the first draft version.

On Thu, Mar 28, 2019 at 12:50 PM Sean Gillies <sean.gillies@...> wrote:
Hi all,

I intend to write a RFC concerning the breaking changes I proposed for Fiona 2.0. Where would these RFC be maintained and what would they look like? I am propose a template here:


Please comment on that PR if you would like to be part of a team that would review future RFC PRs. I hesitate to call this a steering council because I don't want to overdo the governance for a relatively small project, but that's close to what the RFC team would be.

--
Sean Gillies


--
Sean Gillies

1 - 20 of 31