Commit Graph

209 Commits (a4b79ca61014bb5f47296c51bfe54b3aac708763)

Author SHA1 Message Date
Nutomic e9e76549a8
Split activity table into sent and received parts (fixes #3103) (#3583)
* Split activity table into sent and received parts (fixes #3103)

The received activities are only stored in order to avoid processing
the same incoming activity multiple times. For this purpose it is
completely unnecessary to store the data. So we can split the
table into sent_activity and received_activity parts, where
only sent_activity table needs to store activity data. This should
reduce storage use significantly.

Also reduces activity storage duration to three months, we can reduce
this further if necessary.

Additionally the id columns of activity tables are removed because
they are completely unused and risk overflowing (fixes #3560).

* address review

* move insert_received_activity() methods to verify handlers

* remove unnecessary conflict line

* clippy

* use on conflict, add tests
1 year ago
Nutomic 7d8cb93b53
Check for dead federated instances (fixes #2221) (#3427)
* Check for dead federated instances (fixes #2221)

* move to apub crate, use timestamp

* make it compile

* clippy

* use moka to cache blocklists, dead instances, restore orig scheduled tasks

* remove leftover last_alive var

* error handling

* wip

* fix alive check for instances without nodeinfo, add coalesce

* clippy

* move federation blocklist cache to #3486

* unused deps
1 year ago
dullbananas 1d38aad9d3
Make functions work with both connection and pool (#3420)
* a lot

* merge

* Fix stuff broken by merge

* Get rid of repetitive `&mut *context.conn().await?`

* Add blank lines under each line with `conn =`

* Fix style mistakes (partial)

* Revert "Fix style mistakes (partial)"

This reverts commit 48a033b87f.

* Revert "Add blank lines under each line with `conn =`"

This reverts commit 773a6d3beb.

* Revert "Get rid of repetitive `&mut *context.conn().await?`"

This reverts commit d2c6263ea1.

* Use DbConn for CaptchaAnswer methods

* DbConn trait

* Remove more `&mut *`

* Fix stuff

* Re-run CI

* try to make ci start

* fix

* fix

* Fix api_common::utils

* Fix apub::activities::block

* Fix apub::api::resolve_object

* Fix some things

* Revert "Fix some things"

This reverts commit 2bf8574bc8.

* Revert "Fix apub::api::resolve_object"

This reverts commit 3e4059aabb.

* Revert "Fix apub::activities::block"

This reverts commit 3b02389abd.

* Revert "Fix api_common::utils"

This reverts commit 7dc73de613.

* Revert "Revert "Fix api_common::utils""

This reverts commit f740f115e5.

* Revert "Revert "Fix apub::activities::block""

This reverts commit 2ee206af7c.

* Revert "Revert "Fix apub::api::resolve_object""

This reverts commit 96ed8bf2e9.

* Fix fetch_local_site_data

* Fix get_comment_parent_creator

* Remove unused perma deleted text

* Fix routes::feeds

* Fix lib.rs

* Update lib.rs

* rerun ci

* Attempt to create custom GetConn and RunQueryDsl traits

* Start over

* Add GetConn trait

* aaaa

* Revert "aaaa"

This reverts commit acc9ca1aed.

* Revert "Revert "aaaa""

This reverts commit 443a2a00a5.

* still aaaaaaaaaaaaa

* Return to earlier thing

Revert "Add GetConn trait"

This reverts commit ab4e94aea5.

* Try to use DbPool enum

* Revert "Try to use DbPool enum"

This reverts commit e4d1712646.

* DbConn and DbPool enums (db_schema only fails to compile for tests)

* fmt

* Make functions take `&mut DbPool<'_>` and make db_schema tests compile

* Add try_join_with_pool macro and run fix-clippy on more crates

* Fix some errors

* I did it

* Remove function variants that take connection

* rerun ci

* rerun ci

* rerun ci
1 year ago
Nutomic 93225e5ddf
Error enum fixed (#3487)
* Create error type enum

* Replace magic string slices with LemmyErrorTypes

* Remove unused enum

* Add rename snake case to error enum

* Rename functions

* clippy

* Fix merge errors

* Serialize in PascalCase instead of snake_case

* Revert src/lib

* Add serialization tests

* Update translations

* Fix compilation error in test

* Fix another compilation error

* Add code for generating typescript types

* Various fixes to avoid breaking api

* impl From<LemmyErrorType> for LemmyError

* with_lemmy_type

* trigger ci

---------

Co-authored-by: SleeplessOne1917 <abias1122@gmail.com>
1 year ago
Nutomic 3578dab67f
Remove PerformApub trait (#3423)
* Remove PerformApub trait

This is completely useless now that websocket is gone. In the future
I also plan to remove Perform and PerformCrud traits, but it will be
difficult to do that while still compiling crates in parallel.

* params need to use query
1 year ago
dullbananas bef76630c5
Remove redundant calls to `Iterator::collect` (#3365)
* Remove redundant calls to `Iterator::collect`

* Update mentions.rs

* Add clippy lints and run fmt

* CI ran on the wrong commit again 
1 year ago
Sander Saarend e23621c2cd
Calculate initial hot_rank and hot_rank_active for posts and comments from other instances (#3131)
* Calculate initial hot_rank when receiving posts and comments from other instances

* Move hot rank update logic into db_schema
1 year ago
Nutomic 2423b89ced
Activitypub breaking changes for Lemmy 0.18 (#2705)
* Remove deprecated field `group.moderators`

* Drop column local_site.federation_debug

* Remove backwards compatibility code for feature/lock post

https://github.com/LemmyNet/lemmy/pull/2716

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
1 year ago
Nutomic 3565ad984a
Remove chatserver (#2919)
* Remove chatserver

* fix clippy

* Remove captchas (fixes #2922)

* fix prettier

* fix api_common build

* cargo fmt
1 year ago
Nutomic a5707328cf
Dont return error in case optional auth is invalid (#2879)
* Dont return error in case optional auth is invalid

* fixes
1 year ago
Nutomic a25faf3e97
Add `to` field in follow activities for better compatibility (#2829)
* Add `to` field in follow activities for better compatibility (fixes #2744)

* fix tests
1 year ago
Dessalines 33ba1fc623
Separate federated_instances into its own endpoint. Fixes #1931 (#2813) 1 year ago
Dessalines 63f54a3103
Making the chat server an actor. (#2793)
* Making the chat server an actor.

- Fixes #2778
- #2787

* Forgot to add handlers folder.

* Some cleanup.

* Forgot to remove a comment.

* Address PR comments.

* Using ToString for enum operations.
1 year ago
Nutomic 6f513793cb
Activitypub crate rewrite (#2782)
* update activitypub-federation crate to 0.4.0

* fixes

* apub compiles!

* everything compiling!

* almost done, federated follow failing

* some test fixes

* use release

* add code back in
2 years ago
Anon 6bc49bdd70
Add Custom Emojis Support (#2616)
* Add Custom Emojis

* Modify index
2 years ago
Dessalines 48f187188b
Show deleted and removed posts for profile views. Fixes #2624 (#2729)
* Show deleted and removed posts for profile views. Fixes #2624

* Only showing non-deleted/removed posts for creator.

* Add a admin or mod check to views, to show deleted and removed posts.

- Also removed the pointless "blanking" functions

* Fix clippy

* Make hidden posts comment clearer.

* Fixing federation tests.

* Fixing fmt.
2 years ago
Nutomic a42f7271e6
Proper permission check for federated comment distinguish (#2697)
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2 years ago
Nutomic 62663a9f2e
Implement separate mod activities for feature, lock post (#2716)
* Implement separate mod activities for feature, lock post

Also includes collection for featured posts. Later we also need
to do the same for Comment.distinguished

* some changes

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2 years ago
Dessalines 1917e3d495
Separate comment distinguish (#2740)
* Combine prod and dev docker setups using build-arg

- Fixes #2603

* Dont use cache for release build.

* Separate comment distinguish into its own action.

- Fixes #2708

---------

Co-authored-by: Nutomic <me@nutomic.com>
2 years ago
Dessalines 36aaa50644
Organize utils into separate files. Fixes #2295 (#2736)
* Organize utils into separate files. Fixes #2295

* Moving tests.

* Fix test.

* Fix test 2
2 years ago
Dessalines 76e0ab934f
Fixing missing forms, incorrect user discussion_languages (#2580)
* Fixing missing forms, incorrect user discussion_languages

* Add discussion_language to CommunityResponse

* Adding discussion_languages to CommunityResponse

* Adding discussion_language logic to community.create
2 years ago
Anon 9dfd819691
Add support for Featured Posts (#2585)
* Add support for Featured Posts

* Fix rebase

* More fixes
2 years ago
Nutomic 2732a5bf07
Rework websocket (#2598)
* Merge websocket crate into api_common

* Add SendActivity trait so that api crates compile in parallel with lemmy_apub

* Rework websocket code

* fix websocket heartbeat
2 years ago
Felix Ableitner c6c52ab9cc Add SendActivity trait so that api crates compile in parallel with lemmy_apub 2 years ago
Felix Ableitner 201fa97769 Move code to generate apub urls into lemmy_api_common 2 years ago
Felix Ableitner 030afbc2e7 Merge websocket crate into api_common 2 years ago
Nutomic bcf5c91f81
Use audience field to federate items in groups (fixes #2464) (#2584) 2 years ago
Nutomic d20d2b9218
Implement federated user following (fixes #752) (#2577)
* Implement federated user following (fixes #752)

* rewrite send_activity_in_community and add docs, remove default for column pending

* improve migration

* replace null values in db migration
2 years ago
Nutomic 63c237d6df
Use enum_delegate crate (#2554) 2 years ago
Felix Ableitner 0b86ffbdbd Remove unwrap 2 years ago
Nutomic a0fed24cee
When announcing incoming activities, keep extra fields (#2550) 2 years ago
Dessalines 5d837780f5
Add diesel_async, get rid of blocking function (#2510)
* Moving settings to Database.

- Moves many settings into the database. Fixes #2285
- Adds a local_site and instance table. Fixes #2365 . Fixes #2368
- Separates SQL update an insert forms, to avoid runtime errors.
- Adds TypedBuilder to all the SQL forms, instead of default.

* Fix weird clippy issue.

* Removing extra lines.

* Some fixes from suggestions.

* Fixing apub tests.

* Using instance creation helper function.

* Move forms to their own line.

* Trying to fix local_site_data, still broken.

* Testing out async

* Testing out async 2

* Fixing federation tests.

* Trying to fix check features 1.

* Starting on adding diesel async. 1/4th done.

* Added async to views and schema.

* Adding some more async

* Compiling now.

* Added diesel async. Fixes #2465

* Running clippy --fix

* Trying to fix cargo test on drone.

* Trying new muslrust.

* Trying a custom dns

* Trying a custom dns 2

* Trying a custom dns 3

* Trying a custom dns 4

* Trying a custom dns 5

* Trying a custom dns 6

* Trying a custom dns 7

* Addressing PR comments.

* Adding check_apub to all verify functions.

* Reverting back drone.

* Fixing merge

* Fix docker images.

* Adding missing discussion_languages.

* Trying to fix federation tests.

* Fix site setup user creation.

* Fix clippy

* Fix clippy 2

* Test api faster

* Try to fix 1

* Try to fix 2

* What are these lines about

* Trying to fix 3

* Moving federation test back to top.

* Remove logging cat.
2 years ago
Dessalines cb753b045f
Group imports dess (#2526)
* Group imports with rustfmt

* Running cargo fmt again.

Co-authored-by: Felix Ableitner <me@nutomic.com>
2 years ago
Nutomic b0c3185663
Make verify apub url function async (#2514)
* Make verify apub url function async

* cleanup

* use dep from crates.io

* dont use unwrap
2 years ago
Dessalines 235cc8b228
Moving settings to Database. (#2492)
* Moving settings to Database.

- Moves many settings into the database. Fixes #2285
- Adds a local_site and instance table. Fixes #2365 . Fixes #2368
- Separates SQL update an insert forms, to avoid runtime errors.
- Adds TypedBuilder to all the SQL forms, instead of default.

* Fix weird clippy issue.

* Removing extra lines.

* Some fixes from suggestions.

* Fixing apub tests.

* Using instance creation helper function.

* Move forms to their own line.

* Trying to fix local_site_data, still broken.

* Fixing federation tests.

* Trying to fix check features 1.

* Addressing PR comments.

* Adding check_apub to all verify functions.
2 years ago
Nutomic aaaea2006c
Federate votes as private (#2501) 2 years ago
Nutomic 74bc1198ce
Dont try to send activities if federation is disabled (fixes #2499) (#2500) 2 years ago
Nutomic ec5e63b5a9
Fix check for federated mod actions (#2489) 2 years ago
Nutomic 2ef0f8f5f8
implement language tags for site/community in db and api (#2434)
* implement language tags for site/community in db and api

* add api checks for valid languages

* during db migration, update existing users, sites, communities to have all languages enabled

* init new users/communities with site languages (not all languages)

* federate site/community languages

* fix tests

* when updating site languages, limit community languages to this subset

also, when making a new post and subset of user lang, community lang
contains only one item, use that as post lang

* add tests for actor_language db functions

* include language list in siteview/communityview

* Fix some of the review comments

* Some more review changes

* Add todo about boxed query

* Add default_post_language to GetCommunityResponse
2 years ago
Dessalines c9f1407429
Diesel 2.0.0 upgrade (#2452)
* Initial commit to bump diesel to 2.0.0-rc.0 and see what happens

* Add chrono feature from diesel

* db_schema crate is close to building?

* Upgrade diesel-derive-newtype

* Mostly modifying references to connections to be mutable ones; also used
new way to do migrations as suggested by the migration guide; a lot more
compiles now, though I can't figure out this tricky ToSql issue at the
moment

* Running clippy --fix

* Trying to fix drone clippy 1

* Fix clippy

* Upgrade clux-musl

* Trying to fix drone clippy 2

* Trying to fix drone clippy 3

* Trying to fix drone clippy 5

* Adding diesel table aliases, removing sql view hack. Fixes #2101

Co-authored-by: Steven Chu <stevenc1@gmail.com>
Co-authored-by: Nutomic <me@nutomic.com>
2 years ago
Nutomic a85334c675
Handle Like, Undo/Like activities from Mastodon, add tests (fixes #2378) (#2380) 2 years ago
Dessalines c62671116c
Send websocket message on accepted follow. Fixes #2369 (#2375) 2 years ago
Dessalines a745fa6f43
Remove update and read site config. Fixes #2306 (#2329)
* Remove update and read site config. Fixes #2306

* Removing lazy_static, removing Settings::get()
2 years ago
Dessalines 4e12e25c59
Adding admin purging of DB items and pictures. #904 #1331 (#1809)
* First pass at adding admin purge. #904 #1331

* Breaking out purge into 4 tables for the 4 purgeable types.

* Using CommunitySafe instead in view

* Fix db_schema features flags.

* Attempting to pass API key.

* Adding pictrs image purging

- Added pictrs_config block, for API_KEY
- Clear out image columns after purging

* Remove the remove_images field from a few of the purge API calls.

* Fix some suggestions by @nutomic.

* Add separate pictrs reqwest client.

* Update defaults.hjson

Co-authored-by: Nutomic <me@nutomic.com>
2 years ago
Nutomic fcaf7a084c
Upgrade activitypub_federation to 0.2.0, add setting federation.debug (#2300) 2 years ago
Nutomic 7804a9d893
Accept private like (#1968) (#2301) 2 years ago
Nutomic 13ca1111c4
Remove 0.15 federation compat code (#2131) 2 years ago
Nutomic a2a594b763
Extract Activitypub logic into separate library (#2288)
* Create example for apub lib

* some rewriting of apub lib

* Add LocalInstance struct for apub lib to avoid using Lemmy Settings

* Move ActorType trait to lemmy_apub, because its not needed in library

* Use reqwest_retry instead of custom impl, dont specify timeout on every send()

* Some improvements to example

* Moved inbox handling to library

* bug fixes

* Move context and serde helpers into library

* wip: example changes

* Add lemmy_utils feature to build only LemmyError

* Rename to activitypub_federation

* Remove lemmy_utils dep from activitypub_federation using generic error type

* Finish activitypub example

* Cleanup and fix tests

* Reorganize library files

* Remove ApubObject.to_tombstone()

* Extract activitypub library into separate git repository
2 years ago
Nutomic 7b86441bab
Federate with Peertube (#2244) 2 years ago
Nutomic 3aa3d75a1e
Add cargo feature for building lemmy_api_common with mininum deps (#2243) 2 years ago
Nutomic c718882479
Accept comments with hashtags from Friendica (#2236) 2 years ago
Nutomic 8337eaefdd
Federate user account deletion (fixes #1284) (#2199) 3 years ago
Nutomic 9ac1f46a2b
Dont federate initial upvote (#2196) 3 years ago
Nutomic f9d563d80a
Add missing mod log entries for federated actions (fixes #1489) (#2198) 3 years ago
Nutomic 65cac21713
Fix verify_mod_action check for remote admin actions (#2190)
* Fix verify_mod_action check for remote admin actions

* fix federation test
3 years ago
Nutomic bfae246734
Migrate towards using page.attachment field for url (ref #2144) (#2182) 3 years ago
Nutomic dfb0938738
GNU social compatibility (#2100)
* Use SourceCompat everywhere (better compat with other software)

* Name field should not be mandatory in Group

* also check page.cc field for community id

* add gnu social tests

* better to use option<sourcecompat>

* update gnu social tests, marked vote as "unlisted"
3 years ago
Nutomic b406342a14
Dont log error if duplicate activity is received (fixes #2146) (#2148) 3 years ago
Nutomic 8112816e99
If viewed actor isnt in db, fetch it from other instance (#2145) 3 years ago
Nutomic 166ec196b0
Rework error handling (fixes #1714) (#2135) 3 years ago
Nutomic 7d61616dff
Revert "Add logging to debug federation issues (ref #2096) (#2099)" (#2130)
This reverts commit e094989a4c.
3 years ago
Nutomic f8fb3b41f0
Reject federated downvotes if downvotes are disabled (fixes #2124) (#2128) 3 years ago
Nutomic df757f28b4
Make delete activities backwards compatible with 0.15 (#2114) 3 years ago
Nutomic e094989a4c
Add logging to debug federation issues (ref #2096) (#2099) 3 years ago
Nutomic 788924d7ff
Merge different delete activities for better compatibility (fixes #2066) (#2073) 3 years ago
Nutomic dd865c5af5
Implement instance actor (#1798)
* Implement instance actor

* wip: make site bans federate

* finish implementation and unit tests for federated bans

* start adding api tests

* fix api test

* remve site from GetCommunityResponse

* only federate site bans originating from user's home instance

* dont expose site.private_key in api
3 years ago
Dessalines e65c45f152
Adding temporary bans. Fixes #1423 (#1999)
* Adding temporary bans. Fixes #1423

* Adding ban_expires to person

* Fix clippy

* Removing ban_expires from federated fields.

* Trying to add expires to blockcommunity apub.
3 years ago
Aode (lion) 97ebf2f6f3 Consolidate reqwest clients, use reqwest-middleware for tracing 3 years ago
Riley 35cbae61bc
Don't drop error context when adding a message to errors (#1958)
* Respond directly with LemmyError

Instrument Perform implementations for more precise traces
Use ApiError to format JSON errors when messages are present
Keep SpanTrace output in LemmyError Display impl

* Hide SpanTrace debug output from LemmyError

* Don't log when entering spans, only when leaving

* Update actix-web

* Update actix-rt

* Add newline after error info in LemmyError Display impl

* Propogate span information to blocking operations

* Instrument apub functions

* Use skip_all for more instrument attributes, don't skip 'self' in some api actions

* Make message a static string

* Send proper JSON over websocket

* Add 'message' to LemmyError display if present

* Use a quieter root span builder, don't pretty-print logs

* Keep passwords and emails out of logs

* Re-enable logging Login

* Instrument feeds

* Emit our own errors

* Move error log after status code recording

* Make Sensitive generic over the inner type

* Remove line that logged secrets
3 years ago
Felix Ableitner 27a687bcd0 Dont send email notifications for edited comments (fixes #1925) 3 years ago
Riley 3fea5645f8
Add tracing (#1942) 3 years ago
Dessalines 3d08e6c1fc
Adding unique constraint for activity ap_id. Fixes #1878 (#1935)
* Adding unique constraint for activity ap_id. Fixes #1878

* Removing is_activity_already_known
3 years ago
Felix Ableitner afccd5cf69 Replace activitystreams crate with activitystreams-kinds 3 years ago
Felix Ableitner 3e4fe89a45 Fix build error in #1914 3 years ago
Dessalines 6d35508f08
Merge pull request #1914 from LemmyNet/dont-announce-note
Dont announce comments, edited posts to Pleroma/Mastodon followers
3 years ago
Felix Ableitner aeb34199f5 Dont announce comments, edited posts to Pleroma/Mastodon followers 3 years ago
Felix Ableitner a2707e1c56 Community outbox should only contain activities sent by community (fixes #1916) 3 years ago
Felix Ableitner db631005d3 Fix error due to same activity being inserted in database twice 3 years ago
Felix Ableitner 3249060f67 Remove to field from follow activities 3 years ago
Aode (lion) 20cddf5e81 Support mastodon deletes 3 years ago
Felix Ableitner 7f4a773b88 Create and Note always need to tag parent creator, for mastodon notifications 3 years ago
Felix Ableitner c1f1b8aa0f Activities in community should also be sent to actors in cc 3 years ago
Felix Ableitner 579ca37ddf Make webfinger standard compliant 3 years ago
Dessalines 23641fcd06 Use serde_json::to_value 3 years ago
Nutomic 1b9414f292
More federation compat (#1894)
* Make HTTP signatures compatible with Pleroma

* Send Announce/Page, Announce/Note for Pleroma compatibility

* remove unused code
3 years ago
Dessalines 76220a4523
Fixing clippy lints. (#1885)
* Fixing clippy lints.

* Revert object id display

* Trying to fix clippy again
3 years ago
Felix Ableitner 252d87d332 Fix clippy error 3 years ago
Felix Ableitner 888e683856 For verify_is_public() we also need to check cc field 3 years ago
Felix Ableitner a5a674a270 Add method ApubObject.verify() 3 years ago
Felix Ableitner 8ea21c39b7 Reduce stack memory usage in apub code
- use our own, smaller Endpoints struct
- wrap ObjectId.url in Box
- adjust usage of Box in different places
3 years ago
Felix Ableitner c725514841 Change to_apub and from_apub to take by value and avoid cloning 3 years ago
Felix Ableitner 2edf8ba157 Move ObjectId to library 3 years ago
Felix Ableitner bd3352423a Remove ActivityFields trait, deserialize into another struct instead 3 years ago
Felix Ableitner 969a7f2d1b Refactoring apub code 3 years ago
Dessalines a83113935d
Check if post or comment are deleted first. Fixes #1864 (#1867)
* Check if post or comment are deleted first. Fixes #1864

* Refactoring delete apub.

* Revert "Refactoring delete apub."

This reverts commit ba2c3d06cfb870efe792f4b2541036265b425156.
3 years ago
Felix Ableitner b396344eae Merge two functions into one 3 years ago
Felix Ableitner 0bde2d595e Add tests for parsing activities and collections 3 years ago
Felix Ableitner aaaf039779 Move apub test files into tree structure 3 years ago
Felix Ableitner 03d8ac75ef Move activity structs to protocol folder 3 years ago
Felix Ableitner 5ff044346f Move object and collection structs to protocol folder 3 years ago