Skip to content

Conversation

@Ziqi-Li
Copy link
Member

@Ziqi-Li Ziqi-Li commented Feb 25, 2019

Hi, this PR is mainly to solve the memory issue with GWR and MGWR. It may seem a lot of changes. I tried my best to conform with what we have right now. All previous tests are passed. I also added several new test cases. Thanks!

Major changes:

  1. In Kernel class, we used to compute entire spatial weights for all locations. This will need to store W (n by n) in memory. The update will be computing spatial weights on the fly when doing each local regression at location i, which results in not storing entire spatial W matrix. This allows GWR fitting to be applied to large dataset (n >20k).
  2. Add multiprocessing as option for GWR/MGWR calibration. A multiprocessing.Pool object can be passed to Sel_BW.search(pool=pool) and GWR.fit() Example notebook is added.
  3. Change MGWR inference computation from search.multi_bw() to MGWR.fit(). Also, add a new method for computing MGWR inference in chunks by introducing a n_chunks argument in MGWR.fit(n_chunks). e.g. when n_chunks=2 (n_chunks=k), the overall memory usage is reduced by a factor of 2 (k). This allows MGWR fitting to be applied to relatively large dataset (10k ~ 40k) within a reasonable time. The effectiveness in reducing memory by increasing n_chunks can be found here.
  4. Add hat_matrix=False as default option for GWR() and MGWR(). Inference statistics are computed on the fly in each local regression. If entire hat matrix is needed for some reasons, one can specify hat_matrix=True, and then hat matrix can be obtained by GWRResults\MGWRResults.S.

Minor changes:
Bug fixes:

  1. offset and spherical parameters in Set_BW are not passed to gwr_func, thus not in effect. Fixed.
  2. Add several test cases of Sel_BW w/wo offset and w/wo spherical.

Enhancement:

  1. Add adj_R2 for gaussian and D2 and adj_D2 (%of deviance explained) for Binomial and Poisson.
  2. Add tests for adj_R2, D2 and validated against gwr4.

@Ziqi-Li Ziqi-Li requested a review from ljwolf February 25, 2019 23:02
@Ziqi-Li Ziqi-Li marked this pull request as ready for review February 25, 2019 23:03
@Ziqi-Li Ziqi-Li changed the title Memory optimization for GWR/MGWR Memory optimization and Parallelization for GWR/MGWR Mar 19, 2019
@Ziqi-Li Ziqi-Li changed the title Memory optimization and Parallelization for GWR/MGWR Memory optimization and parallelization for GWR/MGWR Mar 19, 2019
@Ziqi-Li Ziqi-Li changed the title Memory optimization and parallelization for GWR/MGWR Memory optimization and optional parallelization for GWR/MGWR Mar 19, 2019
@Ziqi-Li Ziqi-Li merged commit eae8ac3 into pysal:master Mar 19, 2019
@Ziqi-Li
Copy link
Member Author

Ziqi-Li commented Mar 19, 2019

Reviewed with @TaylorOshan and agreed on merging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant